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THE COMMUNICATION OF EXPERIENCE 

•A GUIDEBOOK FOR THE MANAGEMENT OF INFORMATION 
BY U'.S. OFFICE OF EDUCATION ARTS EDUCATION - PROJECT DIRECTORS 

I. INTRODUCTION - ' 

\ This guide is designed to address 'the ^information needs of the Arts 
Educator who functions ^as a manager of a program, project or other^— . 
administratively definable- unit in a local or State -educationaXsystem. 
The material is derived from the experiences of four re'gi onal workshops 
sponsored by the USOE Arts and Humanities Staff\for tteyWZ-lV grant 
•Project Directors. > f l ; 

i 1 

These^guidelines/ provide more than a repoV^f .oftHhe workshops yet 
somewhat less than a detailed "how-to" handbook t 

ih 1 •• ' 

The two-day workshops, entitled The Comrtt/nWation of Experience , 
brought more than 80 .local and State arts e|$caiors together in Boston, 

2 0 f 

Oklahoma City, Phoenix and Indianapolis.' tfieffjpcus of the workshops, as 
of this guidebook, was project success and ; /^'ur\|ival- -and how to manage 
information in order to ensure it* Major |uk>hdses were put on the 
information arts educators need: • -if j 

for their ownVojects 1 effective 'Management (documentation)*; 



to convince othWs (evaluation); arid f 
to share among their peers (dissemination). 



The underlying rationale for the workshops' was that this? information, 
regardless of how it is used, comes initially i from the experiences of the 
project and staff as they work toward their goals. A major task of each 



projects the transformation of these experiences into information and 
its use to-achlevejuccess. Thus the title, The Comnunication -of 
Experience . " , 

Whi)e there were some presentations of "how others have done things" 
at the workshops, the major concepts were developed experienti ally-that 
is, each Project Director brought certain experiences into the sessions 
which helped shape what they took away from it. This guidebook, 
similarly, is not a transcript of workshop presentations or discussions. 
Rather, it is organized- aroun? a framework of questions which synthesize 
many of the concerns of the Arts Educator who serves ,as a manager .-"To 
the degree you share these concerns, the "answers" suggWe* on these - 
pages may be of value. * 

BACKGROUND ? ^ 

This guide, and the workshops, resulted' from repeated calls for 
assistance from l^cal and state grantees of .the USOE Arts Education • 
Program, as we IT as from Arts Educators elsewhere.; The contractor for 
this effort, Applied- Management Sciences of Silver Spring, Maryland, 
conducted an extensive review of the literature injhis area with 
particular focus on recent national and regional/meetings directly 
concerned with evaluation iu arts education. TheWesults of this review 
are reported; with examples of many of the most Relevant papers, in the 
appendices. 



Additionally, art analysis of, the experiences, of Arts Educators in the 
80 projects funded by USOE \n 1978-79 produced the following inferences, 
upon which rest the particular design of the workshops and of this 
guidebook. ' ■ * - . * 

t First is that most people already know how to turn experiences' 
Bnt°thP« h ln r ormat !? n foF ^ own and others' demons 
• SniiJ^i ! dually associate these skills, with the v 
technical terms "evaluation," "documentation," and 
"dissemination.'! - 1 

§ Sj e n^f« e - faC J° rS constr a- in ing this association is a lifetime 
Sith^S!-' 1 ex P enences in w " ic h People have been burdened 

tit, nrll?*l + 9 t X 0niatl0 ! f0r 0therS; have \ound information 
they provide to others used primary to make Wgments about 

k -9 ■ 



them; and have foUnd-the--informat-ion-they-needed--tnctccessible or 
unavailable at the time it was wanted. -This has produced an 
"affectiy^bias"— or, more simply stated, fear— of putting 
program information on paper. 

4 • A third inference serving as a foundation for this approach 
deals with the movement of information between people and 
programs. Good resource management usually requires that each 
decision-maker view his or her project or unit 'as an independent 
entity. But in reality, the success of any venture is based 
upon the ability to take advantage of the interdependence among 
.elements. And, the use of information to support this 
interdependence has been constrai-ned by the fears cited above, 
as well as the fact thafseldom is any one person responsible 
for interdependence. - — 

• Finally, we have an assumption of why'.a project or unit' would" 
want to collect and use information. The answer we" propose is 
survival, hot research. Unless you get theMnformation you need 
to assure success, you may end up. not needing any information 
for measuring it. Too many good demonstration programs in 
education have ended up as cases of "The Operation was a 
Success:.. hut the Doctor Died." Program survival often requires 
extending your concern beyond your grant- period, on to the 
integration of.your results with the ongoing processes of your 
or another's organization. 

ORGANIZATION ' , 

This guidebook is-orga^'ized "backwards." Although information 
processes usually end with decisions, , we will start at that point, that 
is: the transformation of your project's experiences 4 into information 
for use in the successful accomplishment of your purposes, can be 
displayed as: 1 . 



Ekperiences 



-produce— > 



Information ~w>foc/i > 



* "raw" 



Decisions 



"analyzed" * 

"judged" 

"evaluated" 
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However, the planning for' this process- has to start at the other end— with 
consideration of those decisions which are to 6e influenced by your • , 
project's experiences. . . • r 



Decisions 

• what decisions 

• who makes them 



Information 

• ■ is needed 

• jn what form 

• how to get it 



-vokich 
mil be 
gcunzd 
faiom — > 



Experiences 

• of prQject 

• of 'staff 

• of clients 



JHerefore, following a listing of definitions, the first section of 
this guidebook addresses people, decisions and the information they 
need. ~This is followed By sections on reporting and disseminating ) 
information to affect decisions; and finally, documenting your project's 
experiences and evaluating them. The appendices include an array of 
articles and reports which were used as supplementary materials in the 
workshops. ' 



DEFINITIONS 



r Foil owing- are a number of terms that will be used throughout the /" 
: remainder of this guidebook^Ndte that these definitions are intended 
v only to be what we mean by then}'. *' - *• 



,7* 



V 



Assessment : 

i 

V*. 

Decisions : 
Disseminating ; 

Dogumentinq : 

Evaluating : • 



Looking at the conditions existing in an 
environment— resources and peopVe (both clients 
and providers). 

Judgments based \jpon cumulative experience . 

Transforming your own experiences- inta useful 
information for uthe&s. - 1 



Collectingnnformati-on about what you are doing ' 
and learning . ^ 

Looking at what you did as it affects clients, . 
staff, and environment'-- and making a judgment as 
to its worth for you and others. * 

External Evaluator: A person who functions in the role of evaluator 
, who is not connected with the day-to-day 

n operations of the prografn. 

Formative Evaluation : The process of reviewing and evaluating ongoing 

programs for the purpose of .changing strategies 
and mak^rtig improvements^ before it*is too ,late. 

Internal Evaluator : A person or the program dr project staff who 
functions in the role of evaluator. 



Linkage : 



A relationship between two persons/organizations 
(with reference, to .a mutual concern) that serves 
the needs' of both parties. 



Minimum Reporting Requirements : The convenience of compliance 
~ ; without th| discomfort of thought. 



Model : 
Plan : 

Response Burden : 



The way someone else did it.. 

Your best estimate of what should be done,, until 
you get enough new data to guess again. 

The deference between a Federal or State 
decision-maker's desire to have good t information 
and the practitioners desire to provide it. 

An approach to reaching a desired end that allows 
for unanticipated happenings. 

Summative Evaluation ; Looking at the- program upon compl.etion of the 

effort. 



Strategy : 
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II. INFORMATION: Who has it? Who wants it? For what? 

Control and Influence . . - ;§M\' 

Project management is usually coosidered to include: budget im^' 
planning and scheduling, personnel management, development of supWt, 
progress monitoring, evaluation «and dissemination. More simpTy seated, 
this means that there are justf.four elements that you manage— money,* 
time, people-, and information. . 

As managership of these dp you ,f eel you can control? or . influences- 
Control I Influence ? 
Funds. , 

. , Time - • ' ^ f - 

» — ^— ^ . 

People ' * . . , . 

' - — a % „ 

Information v * y 



Most project directors are aware that only three of those elements 
can he "control led. "j Two 1 of them— money and time— are the focus of most 
managemer/t technip^jwi third— information-^can only 6e controlled if 
you have'lt . The f ourth-rpeo.pl e— can't be "controlled'' at alj, but they 
'can be influenced . And it is those two elements— information and ■ 
people— that are closely intertwined in the management of any' . / j 
"\ demonstration or other type of project," - Identifying, generating and 
f using" information to Influence people can be a major factor in the 
J successful integration of arts education into the school system of your 
community and state. ' ' 

:< * tl Sof1shdata/''tiard M data 0 '-' 

• ■ r ■ ■ — " — • , ' ^ ' 

But what types of information are most influential? Ask yourself 

what forms of information most influenced the decisions you made in 
. developing your current project? And, judging by the following article, 

you are probably not alone if you said that your decisions are^most often 

influenced by direct or indirect experiences rather than reports and 

'evaluations. . " ., 
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* What Counts with Congress ? ■ „ v < 

When Congress sets out to make major policy decisions, the most 
important variable, is the personal values of six to 10 congressmen 
and .their staffs,- a top congressional aide told education researcher, 
gtfth&red in San Francisco this* week. ■ 

• In a candid discussion before a sparsely attended session orv , 
" "Designing Educational Research - to«Reflect Federal Policy* Needs," 
Minority Staff Director Robert Andringa of the House^dutation and 
Labor Committee made no bones about the gulf that slffcrates policy, 
makers and'policy researchers. After a little impromptu research of 
his own, Andringa sa'id he came up with this rank-ojdered list of what 
counts mbst heavily in the congressional decision/making process on 
issues such as education: 

(1) personal values of six to 10 members of Congress and the.ir 
staff, >. h ' • ' 

'(2) strong views of respected and trusted friends, 




(3) 
(4) 

^(5) 

(6) 
(7) 
(8) 
• (9) 

ill) 



assumptions about the budget, * 

public opinion and the popular^media, , 

consensus among majdf interest grdups, such as education 
associations, _ 



program data; (descriptive data about #ho is -served, etc.), 
GAO studies. » 
the hearing^Srocess,' ^ 
policy research 

administration- views and lobbying, and 

? c 
program evaluation. 



Educators*. are generally afiiong the least effective witnesses on 5 
Capitol Hill* Andringa told the participants .attending the panel 
session of the American Educational Research Association convention. 
"Evaluations tend to be untimely or toa long or unreadable or in the 
wrong format," he said, ur*gin<) his , listened to "use* bar graphs and 
pie^char-tsV 1 Andringa said the researchers often seem more intent on 
infpressingHheir colleagues than on helping lay people make wise 

decisions. t < * ■ 

* * » * • * * * 

j. (From Education Daily; April 23, 1976) 
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The point is that decision-makers* unlike computers, are most 

influenced 'by their experiences and* fee lings (or views', assumptions, 

etc.), and the feelings of people .around them, than they are by 

statisticsT^o your goal as manager' and evaluator of your project is to. 

gather and present information in a way which affects the way people 

think and feel, » that decisions they make w-ill be in the direction 

which you think best. • *, , 

\ « i 

■ f The reason we call this' process aval uat ion , and not propaganda , is ' 

because the ^foundatibn of your effort! is the gathering of honest 

informationr-not the Vopagation of Hies. But it is critical for you to 

realize that the effects • of teva 1 uat i tin and propaganda are the are the 

sametrthey influence/people's decisions. , . •' 

On the n6xt two pages are some questions that you- and your staff may • 
find useful to identify tije various persons who influence,- or ''are • 
influenced by, your project. •' r 

> ' • « .* ' ' • * ' 



"Statistics are no Substitute for judgement. 
. _ --Hztuiy Clay ■ 
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If your project were completed, what tangible services and/or 
products would exist or have been provided? Who would use or 
have used each product or service? Who would be ultimately 
affected by each product or service? 

Praducts/Serjvices User Ultimate Jar get 



Xa what .extent have the users .or ultimate targets of your 
services been involved in your project? 

...in proposal development? { ^ - 

.-./in a project advisory role? 

T ' — l 

...in providing other feedback? 



How do you use the information you get from them? Do they know 
this? ' 

Who are t*ie people who are totally committed to the "success of 
your project? 



on your staff? * 

in your organization? "T 
ofitside your, organization? 



Who might like you to be successful because ifrwill facilitate 
some goal of theirs? What are their goals or purposes to which, 
you relate? - ; 

Dp you know what you and each staff member "wants" out of the 
project? " • • { . 

How do you see your role as contributing to the success of 
the project? • ' * • • \ ' 

Do others on. the staff know this? 
How will you know you are being successful? 

Who has the information you need? 



Who has information you need in order to be successful? 
about the problem or need you are addressing? > ' 



about the relevance of y^ur product or service?! 



: 1 ! 

about the effectl veness of your product .or service? 
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• How often do you need the ihformatiorf from these people? 



Why would they want to share thatt information with you? 



How can you get it? 



• . What information mechanisms are you using? 

• .. Are staff meetings for, giving information or getting 

it? ■ k " 



Are advisory meetings for giving .information or getting 



Do you ask people to fill out forms? How do you feel v when 
you do this? . • ' 



Do they know how the information is going to be 
used? 



Who is going to be' judging "your success? (Audience for your 
evaluation) ' 

USOE? . You? ' • ' Others? \ 

.Your institution or agency? , . Y&uf staffs ; 

Yfcur community? . , .Your professional Veers? 



What will each audience be looking at? your services. or 
products? your project processes? - • 

Audience Serv.ices/Products / Project Process 



• Upon what information will they be basing their judgments? 

• When will 'they want the information? At the end of the 
project? Throughout, the project? Quarterly? Other times? 

• Why does each audience need the information? 

EC - _. . ' 

'hi ■*!■ r $& r ~' : : 1 



Information irr^Project Problem=So1ving ( 

Your task j|1r> managing the giving and receiving of information would 
be relatively simple if it were not for one factor, ( For some reason, 
when', we approach the varying components of management— planning, 
operating, evaluating— many of us appear to forget miich of what we do 
with information in other aspects of living* Almost mindlessly we as'sume 
that -a different form of behavior is required. We begin to restrict 
information to symbols that can be recorded on paper ; and tend to deal 
with it as an end in itself, apart from the impact it has on the 
experiences of those who receive it and give it. • 

AS an example of how this cultural blindness limits the information 
we can get, for our management purpose, and which we can give to others 
for theirs, we might first look at the simple task of planning . We tend 
to deal with planning as a "straight line 11 process. That is, we have a 
starting point, a goal, and a series of intermediate "objectives to 
accomplish it: 

i 



Start 


> 


Objective 
One 


t 


Objective 
Two • 




1 

GOAL 




> 


> 



When, in the course of program operation, we reach the time that the 
first objective was to have been accomplished, we check to see if vie have 
reached it; and if we have not, we label this a "discrepancy" and try to 
figure put how to achieve the intended objective: 






i - 


PUnned 










start 


> 


objective 




objective 




GOAL 






one ■ < 




- two 







cLUcAzpancy, 
£ IzadU to 
nejA) plan 
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'This may seem logical, but think for a moment about how planning 
works in. a. task such as navigation. Here, too, the navigator begins with 
a "straight line" plan .... " ' .* ' 



position 
one 




position 
two 


i 

— -> 


GOAL 



...but when the -first position, or check-point is reached, and he or she 
discovers 'that unanticipated wind, currents, wrecks, etc., have taken the 
ship off* its -planned, course— a new course (or plan) is developed— but not 
to the old check point. Instead, the course is planned to the original 
goal from the point at which the. person now is: \ 

\ ■ • \ 




position 
one 



position 
two. 



GOAL 




_f«tgdT_ 

location.,, 



This operation continues as the navigator continues to take sightings ; 
every few hours to determine where the^ship actually )s, and to re-plan 
its course. . 




In looking at the, actual zig-zag. course that brings the« navigator to 
the goal, it is possible to suggest that this planning process is based" 
upon a different set of assumptions than those we use in moit » t 
organizations. For. exampie, p appears that: 



• -Knowing where you are is more important than knowing' where 1 you 
thought you'd be. 



K 
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There will always be unanticipated events or jbther things over, 
which you will have no control. V 

• Discrepancy information is positive data that first tells you 
where you are relative to where you want to be, and. then can 
help you identify forces that can be harnessed to achieve your 
goal. . 

• The more frequent the checking (evaluating your position) the 
less chance of devlatTmpFoo far off course. 

Interestingly, all of these assumptions can be equally true in project 
management, yet they are seldom u^ed. One reason for this may be 
"fear"" fear of the discrepancy, of beiny judged wrong, of not achieving 
expectations. Unfortunately, the^loss, that results from this fear is one 
that affects more than the project itself . For tife Information inherent « 

in the discrepancy— -inhere we are, how di.d we get here* what happened that 

* * 

was unanticipated, etc. --is in effect the description of an experience . 
And experiences are not merely judged good or bad, and then passed over. 
<0ne can learn from either one.. When this information, however, Is not 
gathered and shared because of "fear," then little'br no learning takes 
place. . . 



,THE ROAD TO WISDOM 

The road to wisdom? -/Well, it's plain. i 
and simple to express: 

Err 

and err 

and eft again 

but less 

and less 

and less. 
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- A second example of the' limits, we impose on, ourselves by our 

perceptions of management ts evident .when- we try to use an organization 

chart "as a way to i dent Tfy^iWfomStToFnfieeds and/or channels for 
communicating' information, ' • 




These very familiar "maps," existing if not on our walls then at 
least in our minds, portray what we believe to be the relationships in afT^ 
organization. ■ And they do, but only one type of relationship—those that 
relate to decisions that control resources Coming into the organization.. / 
The' "higher" in the pyramid, the larger* the amount of resources affected — ' 
by a decision-maker. • . * 



Incoming 



\ $5oo r 



j '$500 I 



1 • $250 1 



1 $250 1 



| $250 | ' 



Organization charts, however,. are difficult to use for portraying how 
decision-makers relate to each other in terms of the outcomes of. the 
organization— which in most human service organizations involve some type 
of influence on another person. But if we 'think, of -an organization chart 
as a form of map, then we see that it can be re~-drawn without necessarily 
changing" reality— just as mercator and polar .projections portray the. same 



reality while appearing to be different—the difference resulting from 
their being drawn to different reference points— the equator and the 
. ' * poles, < 

fn the same way, if you re-map your organization 1n terms of a 
different. reference point— that 1s, to portray how peoples 1 decisions 
relate to or affect the outcomes you desire— you maty be surprised at the 
number of potential allies and supfftrters for your project that you * 
Identify, as well as the opportunities you perceive for building links, 
with Information , to these others who share a mutual Interest in your 
project's outcomes* ' 

* 

The following "map" is onfc way to portray these relationships in a 
typical school situation. It has> these characteristics: 

t ' each decision-maker -'in the total* organization is displayed with 
reference to the student; yet the traditional "resource" ' 
relationships are not changed; 

t the "space" that each defcls ion-maker primarily Influences with 
his or hfer resource defcis1ons--e.g., classroom, <bu1ld1ng-r1s 
shown 1n a way th'atjncludes the "spaces" of others that are 
Influenced, by those decisions;'*- 

t' each "space" also coincides- "to a great extent with that person's 
^ " job-related sense-of-self . This is where each decision-maker 

strives each day to meet his or her heeds for a sense of worth, 
** ' achievement, power, etc. In other words, within this space 1s 

where the element of job satisfaction 1s achieved or 1s 

potentially available; * * 

t those professionals who are not 'in direct contact 'with students 
or teachers, such as curriculum devfeljopers* are arrar\ged # on a 
line of influence i that relates them $p the outcomes they support 
but clearly shows that they do not snare 1n the "power" of 
controlling the major rasdurc^js of we organization; 

• other groups or Individuals outside "the school whith relate to 
the outcome can be shown. These can include tfcje home,' community 
organizations, other projects within ttie school system, etc. 

\ 
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OUTCOME 

INFLUENCING -- 
DECISIONS 



RESOURCE 
CONTROL— * 
DECISIONS ' . 




.LINE OF 
OPERATIONAL 
OEC1SION-MAKING 
RESPONSIBILITY 



■ LINE OF 
SUPPORT 



Using a format such as this to plot the relationships of the persons 

w\thin your project provides several advantages: 

• - . - * 

• You can identify those individuals who need information from you 
to, accomplish their purposes, or from whom you need information 
to accomplish yours. In your project what information is this? 
When, or how often,' is it needed? 

. • You can deal openly with the fact that each person in a program 
has dual motivation— effective /service and self-interest. The 
knowledge that self-interest intrudes on almost every decision. 
• is r increasingly being acknowledged in research, and in fact, 
satisfaction is now identified as' a key element in the success 
of any program. What types of information would contribute to a 
sense of satisfaction for you? For your staff? 

• If- your program exists on the line of "influence" rather than 
"resource control" you may be able to see that attempts to 

• achieve outcomes at the student level without involving those 
who make the everyday resouree-decisions are bound for eventual 

• failure. You may want to draw lines to those decision-makers 
for whose problem-solving decisions you could offer support. 
What types of problemsido they have to deal with? How does what 
you offer relate to these problems? 

• You can draw lines to those other agencies or individuals 
outside your organization that share interest in your outcomes . 
What types of information do they have that could support your 
objectives-? How can they be involve d so that you have access to 
them? 
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Within a goal-centered perspective such as .this, you can iegin to 
identify the various sources of the information, you need to document or 
gather for achieving your project's objectives. You c&i also identify f 
• the information you have which could be useful to others in accomplishing 
their purposes.' This latter information is the "content" of what is 
usually called evaluation or dissemination. 




/ 



"It is not true, as a good many industrial 
psychologists assert,^, that .human nature resists 
*change. On the contrary, no being in heaven or 
earth is greedier for new things. But there are 
conditions. • • : 

r. the change must appear an improvement; 

• it must not be so rapid or so great as 
to obliterate the psychological land- 
marks which make a man feel at home; 

• nor obliterate his understanding of his 
work, his relations to his fellow-workers, 
his concepts of skill, prestige and 

" social standing in certain jobs. 11 
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HI/ REPORTING AND DISSEMINATION 
Reporting 1 

Before you start thinking about any required reports you 'have tp 
produce as part of jfour project* grant, are you sure you know what is 
expected from you? It was found at the workshop that many Project 
Directors believe that more is required by the funding agency than 
'actually is. For example, consider the following excerpt from the i 
Evaluation Monitoring and Reporting Requirements of the DHEW/USOE General 

Provisions: • { 

^ i 



§100*276 Evaluation* * ' 

-Each project shall Include procedures 
for effective evaluation' of the extent to 
which project objectives are being met. 

(20TT.S.C.1221Cfl»(X).) 



Subpart Q— Monitoring and Reporting of 
Program Perform anc* 

§ lOOa.430 Scop* of subpart. 

This subpart sets forth the' procedures 
for mon i t o rin g and reporting program 
performance. These procedures are de- 
\ signed to place greater reliance on re* 
V dpienta to manage the day-to-day oper- 
. ^ffig P of their federally •supported ac- 
tivities. 

- (OMB Ctreuiar No, A-ioa* Attachment L) 
» ^100a.431 Mbnitoringbywcipkntfc 

- ^Sectpient* shall *yyh*fantly "^rdtor 

- the performance under federally-sup- 
ported- activities to assure that adequate 
progress is being made towards achiev- 
ing the goals of the project This review 
shall be made for each function or ac- 
tivity of each-project aa set forth in the 
approved grant application or contract 
document. 



§ 100sv*34 Significant development* be- 
tween scheduled reporting dates* 

Between the "scheduled performance 
reporting dates, events may. occur which 
have significant impact upon the /eder- 
ally-supportedt a<#vity. In such cases, the 
recipient shall inform the Commissioner 
as soon as the following types of condi- 
tions become known f 

(a) Problems, delays, or adverse 'con- 
ditions which will ^nttrltiHy impair the 
Ability to attain the objectives of the 
grant or contract. This disclosure shag 
be accompanied by, a statement of the 
action taken, or contemplated, and any 
Federal assistance needed to resolve the 
situation. L * 

(b) Favorable developments or events 
which enable meeting time schedules and 
goals sooner or at less cost than antici- 
pated or producing more beneficial re* 
suits than originally projected. 



In addition to these general requirements, the USOE Arts Education 
staff have indicated the types of information which are useful .to them, 
as in the following excerpt entitled "Final Reports" taken from a USOE. 
Staff Memo to the 1977 projects: 



The staff at trie national level responsible for administering 

the program will not be able* to visit many, of the -77 State. and local 
projects, much as we would like to. We f learn something of what you 

are doing by talking with you and your associates by telephone and at 

conferences, but we need to add to this information. The/reports 

will help us answer the many requests we receive for details on the 
grants./ . * 
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An informal narrative report will suffice. No specific form, 
need be completed, nor are you required to -supply specific 
statistical data. (However, if such data are available we would 
appreciate your including this information;) 

We would like you, to include in- your report: 

• a brief summary of specific activities the grant supported 

• \ the project's strengths and weaknesses 

• major problems and/or obstacles encountered during the year 

• 'issues (those which emerged during the year as well as 
'those present at the beginning of the project) 

• project outcomes (progress, gains, compromises, 
implications .for the future) { * 

% specific plans -for the future (for arts education in ^pur 
school, district, or State; for* the projector its goals) 

We are interested in v what you feel the project accomplished 
(whether or not these accomplishments relate to the Original 
proposal). We hope you will consider the report an opportimity to 
record your own reflections, intuition, and t feel ings,.about the 
project, as well as the expediences of students, teachers, and 
administrators involved. 

In.addition, we would appreciate copies of the materials which 
the_ grant helped support; i.e., curriculum materials, ^in-service 
tra'ining materials, -comprehensive plans* resourcefcguides. Do not 
send but let us know if you developed slides, films, videotapes, 'or 
other audio-visual materials with grant funds. 
t 

It is obvious from the above that the types of information that 

persons operating at the Federal level most' Reed include many of the 

feelings and learnings that do not fit easily on checklists and one page 

forms. This creates a problem for Federal programs because ^hey.are y 

'constrained from creating "cmdue response burdens 11 on local programs. 

/ - ''V 

•Earlier we defined' response burdens as the "difference between a Federal 

or State, decision-maker's desire to have good information and the 

practitioner's desire to'provide ii." On the one hand, today, the 

awareness of response burden has led to shorter and less frequent report 

forms in many programs. But on the other -hand, it has also slowly cut- * 

off the flow, of learnings and other experiential information that can let 

'Federal personnel play roles more helpful and supportive than just being 

compliance monitors. f , - • 
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One way to dell with this situation has- been suggested in the 
proposed "new regulations for ±he USOE Arts Education Program': ■ 



Subpart E-Whtt Condition* 
MrtbyaQrantM? 

1li1a4J Ongoing lohnUun 



^^JfJsasss sfssa.** ; - • ^r notheracadeaicsub,e " s2gd -" e ' 

' !I ?l»o"f»T atl0n of? 811 "tiviti.s for some Vtne^oraa^o" i^raera'pV fa") 
UJCollection of statistics and data reports. i. 

' .2S y «uST«» needed t0 "»•»» progress (c) Not more that 10 percent of' total 

and develop local support; for example-- project costs may be used for Outside ' 

evaluation of the prafec$^2_ j 
, (^0 U.S.C. 2962) 

Here, in these new regulations, the program is articulating the ' 
^ principle that all -reports are derived from information which, in the 
first instance, must be gathered by the. local project for its own 
management purposes. . « 

Disseminatio n > « 

; : • * 

• Dissemination of information about a project's accomplishments is a 
requirement of most Federal demonstration programs. -It provides a way to 
maximize the investment that the government makes in the development of 
pew ideas. In addition, most project directors want to «j hare information 
about what they are doing, especially with peers and others operating 
similar programs. \ - 

Why then have the results of so many dissemination efforts been 
unsatisfying? . " , 

Th'e idea of experience-sharing among individuals ory projects in a 
Federal program is a deceptively enticing concept that wo^jld work ... 

• If individuals could be aware of what is happening- while it is 
happening. 

. • ' If experience and information were synonymous— that is, if the 
actions and thoughts of individuals, as they dealt with the 
major tasks of operating a '.successful project, -could, be 
translated into a useful arjd, communicable form of information. 

• .11 sharing weren't' left .to. the "good intentions" of parties 

whose intentions soon get overridden by the daily problems of 
running a project or'program. 
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* • If the sharing codld take place- while the problems were current. 

• If individuals felt it was as safe to share the critical 
information about what did not work as about what did. 

• Lf the fear of evaluation and judgment could' be removed from the 
process of providing information to others fn a program, 
especially thpse in higher positions. * - 

We have been and will J)e discussing -ways to identify, and dogument 
this type of information. In this section', however, we will focus ,on 
dissemination as the movement of information among those who have the 
"questions" and those with the "answers." 

i ' * * 

Although this may' be of more, interest for State projects,' or others 
with-the responsibility for maintaining dissemination networks, the 
issues discussed also inflqence the way a specific locaV project can 
package and share information about its worjc. - 

There are three issues that } *ieed to be considered when planning 
dissemination--Why disseminate? What^to disseminate? And how to do it? 

Why Disseminate ?— The Adoption/ Adaptation Issjie 

Leaving aside for a moment the grant requirements to disseminate, it 
may be important for you to clarify for yourself how you feel about why 
dissemination is important. 

TVme ok F&U>z ? 'The reason to disseminate is so that others do not- 
have to re-invent the wheSli 

■ I 

Your answer will. put you somewhere on a continuum between those who 
believe that good*materials and ideas should be adopted without change, 
and those who believe that they must' always be adapted . Or*, said another 
way, between those who believe that the ""wheels' 1 , are more important than 
the process of creating them, and those who believe that the satisfying 
.process of creative adaptation is an important element in teaching and 
the professional growth of all educators. 

Whgtt to Disseminate ?— Tfie What We Did/How We did It Issue + 

Another consideration related, to dissemination relates to the type of 
information most helpful to fit hers. Formal dissemination usually 
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encompasses what might be termed the "WHAT" information— what happened, 
to whom, with what effects? etc. On the other hand, the "HOW'" 
^information— how you .did it, what worked, what didn't work— appears to be 
the information most needed by local programs. This latter information 
'Is the .type that is exchanged, usually informally, through meetings and 
phone calls. Yet most projects do not have ways to systematically 
capture or document this "HOW" information while it is still fresh. (If 
you are interested in one way to addre'ss this problem, see the discussion 
of the Growth Record in the later section about documentation.) 

fry How to Disseminate ?— The Push/Pull Issue 

A final consideration addresse* the issue of how information gets to° 
people.. This might be envisioned as a "PUSH-PULL" continuum. On the one 
end- are those who see the role of dissemination to he the "pushing out" 
of information' to people who are perceived as needing it.. On the other 
end are those who believe that dissemination systems should allow people 
to "pull" or get information only when they need it. 

9 Actually there is«na right position on the three Issues dichotomized 
* absve. Sound' dissemination practice requires that all-sfx of the 
'elements be taken Inter account in any 4 dissemination system. 



A process cannot "btf%nder stood by 

stopping it. 
Understanding must move with the 

flow of the process, must join 

it, and flow with it. 

(First Law of the Mentat) 

Ftiank H&ibeAt 
Vane. 
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Packaging * v ' , 

There are aTso some considerations to be taken Into account when. . 
packaging Information about your project: 

t 

Multiple audiences 

The more different audiences you envision for your material the less 
the chance you wUl communicate effectively with any one of them. If 
possible, develop separate materlalsr-booklets, slide tapes, reports, 
etc.— tail diced specifically to the audience you desire. • Sometimes this 
only Involves the re-ordering of Information ybu already have developed. 
1 , Media 

•Using non-print media for your reports can. have two effects. It can 
. make 1t possible to communicate imaginative and experiential parts of 
your program. On the other hand, it can. limit the access to the 
material. In choosing" the media for. your communication,' consider whether 
or not your desired audience has access to the playback equipment that 
will be needed. „ 

Content ' . « 

— " _— ~~— ■ > - . i • 

The one thing you can be sure you have in common with anyone* who will 
find your experiences useful is that you both are responding to some 
similar problem; e.g., integration of arts into the classroom. 
Therefore, a -useful format many times can v be one that focuses on problems 
and how your project responds $o them. Ask yourself, If someone were 
visiting the project what would interest them the most? Use this in your 
report. • • ■ 

People have different abilities and different experiences which 
influence the way they receive and use final reports. These differences 
- must betaken into account when interpreting findings and reporting 
** information. For' example, reporting achievement tests scores may -tie 1 
meaningful to some audiences, while others may be alienated by the use of 
such .information. * ' 

.<■ ■ 

The primary determination that should be made in the development of a 
^report is to understand the criteria tha^t will be used to judge a program 
or product. 

' . ' . i 
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BeWHTa summary of ideas about reporting that Project Directors 
offered during the course 'of the workshops. You may want to compare them 
to yours. * 

r 
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an abstract 

who was the target group? 

what was the objective? 

what strategy or technique was used? 

What features are different from other projects? 

cost: start-up; special equipment or staff inq needs? 
continuation!? 3 

evidence of effectiveness: 
How was; it tested? • 
On whom'^ 

How. man^ were tested? 



How were they selected? 

measures were they tested?- 

testing: * 



By what 
responses to 



What type of responses occurred? 
At what) frequency? 

Within what range of possible responses?, 
crucial, elements: 

Which did you find essential? 
When die 1 you employ them? s 

possibilities for modification: 

What alternatives were. tested? 

What alternatives are suggested? r 

What problems might be encountered? v 

prepared-materials: 

Clear,.* detailed instructions to primary user 
Separate manuals/instructions for other users 
Project brochure 



I' 



Self -con 
Compatib 



Public information bqoklet(s) 



ained modules 

lity with existing programs or schedules 
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What are the various types of reports that can be considered for 
different audiences ? ! 

written? • * 

oral? 
general? 
specific? 
technical? 
nontechnical? 
descriptive only? 
evaluative and judgmental? 
makes recommendations? 



What Modes of Display Can You Consider ? 



case studie^ 

portrayal s ^ 

graphs and charts 

test score summaries 
scenarios/anecdotes _ 

question's/ answers 

product displays 



dialogues/testimonies ^ 

photographs/newspaper clippings 
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IV. EVALUATION 

What Evaluation Is 

T » 

Evaluation can be defined as looking at what you did as it affects 
clients, staff apd environment— and making a judgment as to its worth for 
you and others. / A breakdown of this definition provides further 
expl anation: 



'/-...looking at what you djd..." 

The first feature of any evaluation is documentation of th> process 
*in an on-going fashion (what has been called "formative evaluation.") 
When documentation is'conducted in this way, a program managerl can make 
assessments at any point in a project's life. The more detailed this 
documentation, the greater the understanding of the process fori both the 
program staff and external observers. , I 

"...as it affects clients, staff, and environment ...V - 

An important feature of any evaluation is the ability to show change 
in tehaviors or attitudes as a rdsult of an innovation. The degree of 
change or effect may be small, but noteworthy. Effects may be dbe to 
many thin/s; the intent here is not to "prove" that what you've done has 
created / change, but rattier provide evidence of differences which. now 
exist i ar the environment since the program or innovation has been 
introdi/ced~so-called "before and after" information. 

"...making a judgment.. ."- 
f ' ' ... A 1 

Evaluation always implies judgment. Whether we like it or notj people 

wilTjudge our projects. As one evaluator, Michael Scriven ('see his 

evaluation model on page 48) has put it, ; 

"...if the goal of evaluation is to determine worth, then, to 
collectdata, focus inquiry and inspect causal relationships 
without making a judgment of worth is not evaluation." I 

This- judgment can be an informed one, based on documented informa- 
tion, or the judgment can be made without benefit of documentation, or of 
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information. When a program is evaluated, we seek to influence these • 
judgments by providing evidence of success, change, and/or differences." 

"...as to its worth- for you and others..." 

*' Evaluation is always subjective. Evaluation is a necessary part of 
the decision-making process, a process wiiich utilizes information as well 
as the values and attitudes of the dectsion-maker.' Another "evaluatdr," 
Laura Chapman, writes that , 

-...the purpose of evaluation 1s to discover the value or 



significance of something. Measurement 1s not an Integral part 
of evaluation; 1t Imposes on the concept of "worth"' an ordering 
and ranking system which leads, to a denial of Intrinsic v^lue. 
An evaluation of an ( arts program should be empirical; valid, 
broadly reliable and responsivejto the qualitative character of 
art experience." ! 

; I * 

But. if it is Important to "sell" a pjrogram to the "outside," one must 
acknowledge the criteria for success ased by such external audiences and 
provide Information (evidence) as to hojf the program has achieved that 
success. - 



t 



Conducting an Evaluation > 

How do you go about conducting an ^valuation? You can conduct an 
evaluation as others have done, using established evaluation "models." 
You can design your own evaluation technique which will satisfy the needs 
of your program,' or you can use a combination of your own model and t.hose 
of others. The important thing is to (select* an evaluation strategy that 
you are comfortable with and which satisfies your own decision-making 
needs. . ri 

* 

Evaluation can assist 1n the decision-making process -for 

^ % program planning 
j c. program improvement , ' 
• 9 program justification. 

- • ^ • ■ ■ 
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When evaluation is used for program planning decis ions it has several 

< i r 

- features. First, evaluation can help to identify needs within the 
program that are contingent upon the ultimate success of a program. Many 
times within the course of running a program, even if a formal needs- 
assessment has been conducted, issues or needs will arise that must be • 
addressed in order to accomplish program objectives. By .establishing a 
formative feedback evaluation system* for decision-making, these needs can 

'be identified early in the program and steps. can generally be taken to 
satisfy these needs. •'" ' 

w Secondly, evaluation can help to identify the discrepancies between 
intended and actual program elements (see the discussion of the • 
Discrepancy Evaluation Model, on page 46). Tfhen a project- is 
conceptualized, we identify outcomes, and the processes necessary to 
achieve these outcomes.' In the course of running a program, we implement 
the processes in the anticipation of the desired results. Because 
programs are made up of people as well as processes, often what we 
suppose will happen, does not, in fact, occur.' Sometimes this *' > . 
realization does not take place until late in the project cycle, or if 
the realization occurs earlier in this cycle, we feel locked-in to our . 
process and unable. fo make necessary changes. Evaluating the process can 
assist here in two ways— i.t can provide early identification of 
discrepancies and it can also provide evidence for the ne'ed for change to 
both internal and external decision-making bodies. 

Finally, evaluation is useful in the planning process; because it can 
serve as*'a checklist to the dec; s ion-maker to assure' that the steps are 
being taken which will lead to project success" (this, again, is what is 
meant by "formative evaluation"). 

Evaluation is also, used in making decisions about program 
improvement . Management is a const ant state of trying to improve and 1 < 
attain greater .levels of satisfaction among staff and constituency. 
Evaluation can be used to develop change strategies and alter the course 
of a program to this end. 

% 
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But evaluation, in the end, is mbst often used for program 

justification . Why should this program continue? When we need to f 

justify a program we seek evaluation information that speaks to consumer 
utility and cost-efficiency. ; , ' . 

Understanding why we conduct an evaluation for planning, .improvement, 
or 'justification >*411 help us in desigping evaluation techniques and 
reporting findings. ; . 

ye 

Steps in Evaluation 

There are four basic steps in conducting an evaluation: 

I. Identification of issues (what needs of the audience must 
be addressed?) \„ • 

II. Identification of information soQrces (documentation) ' 

III. Identification of a strategy (either an "evaluation model" 
or a design of your own) / 

IV Identification of resources X staff and money available for 
conducting the evaluation) , * 

These four steps w^ll be discussed more fully in the sections which 
follow. * 

Identification of- Issues 
(Audience Needs} 

Who is going to be judging your success? 

usoe ; ? • 

Your institution, - > ? 
Your community __• a _J 

You ; > ■ , ? % . 

Your staff ; __? 

Your professional peers _? . , ' ; 

Others I *J 

In designing an evaluation, it 'is Important to understand not only 
why you. are collecting information, but also fo/ whom you are collecting 
information. The audiences of your evaluation^ are those persons and. 
organizations who will be making decisions about. your prftgram. 

What will each audience be looking at? * . 



o Processes— what was it you did? v 
o . Cost— how much? 

o Consequences— how did your project affect your community, or the 
future of your students? ^ 

o Change Factors— what appears to be the reason for changes which 
occurred? 

o Client Satisfaction— did the students like the program? 

o Cognitive or Affective Change— were there changes in what people 
have learned, or in ways in which they feel? 

I Whether or not you want certain groups or organizations to make * ' 
judgments about your program will not prevent such judgments from being 
made. When you undertake the process of evaluation, you, as a program 
manager, have made a decision ta "inform" these judgments. By providing 
your audience with information that is directed to their needs, you gfeve 
them concrete evidence upon which to base a decision. Remember that jOst 
as in a court of law, evidence here can be considered as inadmissible anc 
circumstantial to some, and as concrete proof to others (see the 
discussion of the Jurisprudence Evaluation Model, following on page 54). 
So another feature of an evaluation plan is the determination of what 
evidence will be acceptable to your audience. 

How will you know and prove that your project has been successful? 

* 

, o identify the criteria by which different people make judgments 

o • identify the "political atmosphere" in which the decision-makers, 
operate. 1 



While you and I have lips and voices which -are 

kissing and to sing with 
Who cares if some one-eyed son-of-a-bitch invents 
* an instrument to measure spring with* 

—e.e. cwming* 
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Identification of Information Sources; 
(Documentation) 

Following is a series of questions w/ich deal with the information 
you will need to collect for your evaluation and where this information'" t 
can be found. The sections on docunrentation which follow give a more 
complete description of how to collect and analyze, information,./ 

• What information mechanisms do you use? 

• staff meetings— for* giving, or getting information ? 

advisory meetings— for giving or getting information ? 

Ar> people filling out 'forms' for you? Do they know 



why? \_ ? 

Who has the information you need to be" successful ? 

How often do you need this information from, these 

people . ? 

Why would people want to share this information with 

you . _^ ? 

How can. you get information: 

discussions ? 

observations » ? 

correspondence ' ; ? 

written materials " ? 
v meetings - ? 



It i§ important for the success of. Arts Education programs, and the 
evaluation efforts that parallel them, that documentation and evaluation 
processes not be confused . Regardless of evaluation type or 
strategy— internal or external, project or program, process -or 
product—what is important is. that the information upon which those 
judgments and decisions will be based comes, in the first place, from the 
experiences of the people at local and State levels. Documentation— the 
translating of those experiences into communicable information— is a 
prerequisite for both good management and good evaluatioa. 

It cannot be assumed that information and experience are 
synonymous— that those who are in the middle of an experience are aware 
of what is meaningful in it (and if they were, that they could 
communicate it to others in^ helpful way). Therefore the responsibility 
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for documenting what is. happening and what decisions are being made 
should become a regular function of project. operations, not something you 
"assume" will be done by someone .else, or by yourself, only later 

Documentation might be considered "process observation,", but it is 
important to differentiate between the "process observer" role in 
traditional research, where an individual, working independently, gathers 
"objective" data on what is happening for later external analysis, and 
the process observer role in management. In the latter situation, the 
role involves the whole project team. Using various appropriate methods 
the Project Director; or ario&er individual, increases the awareness of, 
the staff to the experiences which the program is undergoing, why and how 
things are happening, and why various decisions are being made— and this 
person helps the staff .to use the information they^ generate for project 
problem-solving and decision-making. 

An additional and valuable use of this information, of course, is for 
devaluation." While it is critical that Project. Directors and staff have 
access to current information in order to run a project, it is equally 
important that they be able to turn it into timely reports' for different 
audiences of internal and external .decision-makers whose judgments can 
influence the project's success and ultimate existence. 

The task of gatheripj information about the occurrence of events is 
relatively simple compared to that of getting the data about the 
relationships between and among events, and about the, processes that 
precede them. It is not that the information 'does not exist, only that 
it is stored in the "experiences" of those who were the participants. 
They are frequently not aware of all of itf meanings nor of its value to 
ose who* might be looking at the event from another perspective. 

' The nature of this experiential information is what makes it so 
valuable as a target. of the documenting process— it deals with.how and 
Why. things happened. Since the primary interest and emphasis of rthe Arts 
Education Project Directors! Workshops dealt with the "communication of-' 
pperience," the Growth Record Process was presented as one way that 
Project Directors might find useful, to translate regularly the 
experiences of the project participants into useful and meaningful 
information. • 

« ' 
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Growth Record Process : Planning Ahead is, a Looking Backward Process 



As noted earlier, it has been our experience that. there needs v to be a 
distinction made between the role of documentation in research and in 
management. What makes this particularly important are th6 perceptions 
held by many managers that evaluation is an external, process done by and 
for someone else; a process that makes judgments Miich are often based 
upop sketchy or incorrect information about what really happened. 

As evaluators in one federal program noted, 

"Our pervasive impression is that evaluation is not a very well-liked 
part of the program in many-sites ... it still has a slightly" 
unnatural, feel in the base of project operations. It. retains a* 
N flavor of an "add on"— of "something being done to us," It 1s 
extremely difficult to govern and conduct evaluation so that 
participants share in its ownership. 

For these reasons, Applied Management Sciences advocates 
experimenting with an approach to documentation called the Growth Record 
Process. The fundamental concepts in. this approach are: 

• understanding of an experience— what happened, why and how— can 
best be derived aftfer it is over, 

• To do this requires a reference point to focus your awareness- 
something to look back at ^to create a structure for the review^ 
"Planned" objectives serve this purpose Well; for example, "What 
did we intend to do? How? What actually happened? Why? What 
were the influences that affected the outcomes?" 

• The more points of view, or experiences of a situation that are 
considered in the review process, the "better" the picture of 
what happened. This also addresses 'the concern for a 
documentor's objectivity— t)ie problem of self-selection of 
information. The more persons involved, the. more likely. that . 
one person's values won't hopelessly bias the evaluation. 

• This process of identifying what was important is an experience, 
% analyzing -*it, making generalizations and documenting these 

learnings so that they can be used in future actions lets a 
^ staff or board experience the "cth-ha's" .of discoveryjind 
learning that external evaluators usually reserve for 
themselves. This type- of, review should follow each major 
project, activity or event, and be done at least once'a week, 
e.g., at a staff or board meeting,* to pick up other aspects of 
project development. 
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• Participation in this review process provides a way for various 
role groups, or agencies, involved in a project to collaborate 
on a meaningful task;' that is, to contribute to the solution of 
problems that relate to their mutual concern. - The "meeting," 
therefore, as a problem-solving forum, serves as a. major soujpce 
of process data for the documentation. / 

• The frequency (suggested as once a week) of the "process"" 
reviews may seem too high to one who is accustomed to a- culture 
which already seems to, have too many meetings already built-in. 

An analogy for this type meeting, however, c§i be drawn 
, from the process of navigation,, discussed earlier. Evert though 
the original course (plan) is laid out as a. straight line, 
between two points, the navigator is accountable for. continuous 
and frequent checking to determine where they actually are; what 
unanticipated forces caused them to be there; and to suggest a 
^new course "based upon where they are now which takes into 
account the previously unanticipated' influences. 

• • In this process, "knowing where you 'are" is more important 
for planning than knowing where you thought, you would be. « 
Checking frequently is essential because the longer the time 
between check points, the larger the. possible "error," and the 
possibility of forgetting some of the dynamics' of the events. 

The strategies that support this processions ist of a functional role 
for a documentor and self-monitoring "forms" which permit* a project tq 
document its own growth. A discussion of these^two. elements follows. 

The Role of the Documentor \ \ *• 

*f ' ' • ■ ' 

, It is helpful to have one person- accept the responsibility for the 

documenting process. This might be the Project Director, or, it can also 

be a member of the project support staff. \The role of the documentor can 

be perceived, as that of an internal facilitator for the flow of 

information needed for project management. J 

; , ■ • 

Ip a. previous application of this strategy in a national program, 
sucVpersons were called "information facilitators." As described in' ' 
that program, . - * 

his/her principal" taskSis to assure that adequate information is 
.generated and fed back into the decision-making process within the 
project, to "mirror" What- is, going^n, and to facilitate the asking 
of "better" questions. To^da this requires a recognition of this 
role in the project; a sensitivity to the needs and strengths of 
project personnel; and a commitment to. the project's approach. 



KJther important elements of this person's role include serving 
as an interface between the project and external information 
•requirements, i.e., primarily OE through its quarterly reports; and 
serving as a linkage agent to facilitate sharing of experiences among 
projects. -«'.'/• 

v ' Final Report on the Development of' the NDEP 

Information System, OEGrO-70-2068, Sept. 1973. 

The *ole of the documentor responds to one Federal program's 
recommendation "to make evaluation so much an 'integral part of the 
project that it becomes most indistinguishable ... so that information 
^from participants can' be gathered more easily and all project personnel 
can become accustomed to a regular feedback process and to the utiliza- 
tion of data for continuous program improvement." This- documentor role 
helps everyone to be a process evaluator and therefore can remove much of 
the fear of evaluation as an external process. 

The Project Growth Record : A Way to Translate Experie nce- into Useful 
Information 

The pieces of paper that we. call the Project Growth Record are 
designed to provide a project and. a documentor vlith a means for 
generating and collecting experiential information about how a project 
accomplishes its tasks; , what it has to overcome in doing this; what it 
yearns in the process; and how it applies the. learning. It is both a 
planning and a self-reporting system that surfaces problems before they 
become unmanageable, and then challenges project staff to look for 
alternative ways to deal with- the problems. It Is not an "information 
system" but can interface with' whatever formal or informal management * 
information systems. are used in a project. 

The Grovfth Record Process is designed to document the outcomes of a 
project's ongoing decision-making processes, as opposed to a -process, 
which requires that project staff produce and send data to others for 
analysis before getting any value from it. 

Copies of forms follow, with a flow, chart that illustrates how they 
are^sed by project staff to , look back at their month's experience, to 
learn from it, and to re-apply the learnings. 
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PROJECT: 



IttSK/OBJECTIVE: 



Describe the'principal actions or events which took place', 
during this period. 



CO 
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EXPERIENCE 
REPORT 

For the period of: 



What were the observable indicators that each action" 
had the effects yQU anticipated?,., or that it had 
other effects? > 9 
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PROJECT: 



TASK/OBJECTIVE: 



to 
oo 



Ith 




n Comparing your actual EXPER1BCCS 



this period with your plan, what 
[things happened that you Might not 

ive foreseen? 



L 



HINDSIGHT ] 

J ^ 

What actions planned during this period did 
not turn out as expected? Whaf caused the 
difference? / 



1 



ii.iOHTH 



LEARNINGS 



For the Period of: 





SERENDIPITY 



What unexpected favorable 4evelbpmerits occurred 
during this period? % 




P. 



What learnings can you derive from these 
experiences that can be applied to your 
PLAN for the jiext period? ... or can be 
shared with other WEEA grantees? 
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PROJECT: 



TASK/OBJECTIVE: 



7 



POSSIBLE PROBLEMS 
& RESOURCES 



What special OBSTACLES do you anticipate encountering 
during the next period? 




What RESOURCES kre there within your Proj ect for 
dealing with them? * — 



What RESOURCES arJ there outside your Pro tect for 
dealing with them? 1 




HOW do you plan to address the above OBSTACLES? 



J 
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y| • PROJECT: • V- • • 
TASK/OBJECTIVE: ^ 


(ACTION PLAN ^ 

For the period of: 


• 


- J ■ ' - v 


v - • - '-".J 




: 

/ . Miat are the major tasks that you plan to undertake 
this coning period? 

• * " ' ' ' . . 

9 * * 

* / 

f ft * 


At the end of the reriodwhat differences in^v 
Conditions, or in the Actions of people -will ' 
satisfy you that this activity had the effects 
- you-desired? 

1 

h * 

i 

\ 

• 




• 

, • <» 
1 

~ ! 
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2 — • 



VhUft comparing choir «ceu«l 
pcogr«ftft to thftlr origin*! 
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The structure of the forms and the wording of the questions asked can 
vary from- the prototypes shown on the previous pages, but the basic 
principles are usually maintained. The formats provide a way for a 
, project to document its decision processes for activities relatedto each 
major objective. These decisions are perceived as the outcomes of 
problem-solving processes that require experiential information from more 
than one person. 

The decisions (or plans)* are the last step in the process. The focus 
is on looking back at a previous plan (expectation) and comparing it with 
actual accomplishments in order to learn from the positive or negative 
discrepancies. It should be noted that this differs from Management- by- 
Objective-related discrepancy evaluation where discrepancies are 
o "judged", evaluated, and common/y perceived as "bad." Here, instead, 
a discrepancy is defined as an experience (what actually happened and 
why). There is no such thing as a good or bad' experience since it is 
possible to learn from both; and so-called "bad" experiences frequently 
provide more useful information than the "good" ones. It is also 
possible to pick up information about "lucky," or "unexpected" 
occurrences, which wduld be lost in a Management-by-Ob jective reporting, 
because they were never planned for" ; ' ' 

The project documentor, utilizing this type of strategy, serves to 
facilitate his or her projects' ongoing evaluation and planning. 
Moreover, time is a most important dimension of these reports. No single 
report is significant except- in terms of its relationships to earlier or 
later information. It is this "picture" of change or growth over 
. time— and the reasons for it— that provides a "process" picture 
frequently impossible to capture through traditional research or 
evaluation. A periodic (monthly or quarterly) review across a series of 
weekly reports can permit a project to perceive patterns or trends that 
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may be Significant. .Further, it allows them to identify areas of concern 
for additional probing and data collection by the documentor. Thjs 
analysis of activity at quarterly review points can serve to generate a 
great deal of the process (how and why) data needed by Arts Education ' 
project directors. 

One more thing needs td^e said about this process. Since- it is 
based upon both common sense and evaluation-theory, projects may have 
already implemented many of the elements. One of -the attractions for 
project personnel is that they really do jot have to learn something 
new. Many Project Directors with prior management experience already 
will have informal mechanisms that they have used for their own process, 
evaluation. It will tie possible to use those as a starting point or to 
modify these suggested methods and formats 'to relate to their own pre- 
existing procedures. 



V ' 



RLdWid Leak&ti, "There haye been thousands of 
•living organisms", he says, "of which a very 
high percent Has become extinct. There is - 
nothing, at the momftit to suggest that we are 
not part of the same>attern: fle notes that 
there is one point of difference: man is the 
only organism with power to reflect on its 
-past and upon its future. That power to 
reflect, he says, "is what makes us able to 
plan our future in such a way as to avoids 
what seems inevitable." 

• (p. 75 Timz, Mow. 7, J 977) 



Identification of Resources 
(Staff and Money) 

i 

Who will conduct the evaluation ? 

Evaluation can be done by theproject Director or staff member; or by 
someone who is external to the program. Whoever is selected to. do the 
evaluation should have a clearly-defined roTe.. When a member of the ' 
project team is acting as evaluator, that should be known to everyone. 
If dne chooses to use an 'evaluator outside the project, as in the hiring 
of any professional, standard procedures should be used. An examination 
of: , 

• practical background (What do they know about the content and 
context of your program?) 

• educational background (What type of credentials are impdmnt 
to the audience?) 

• • philosophic orierftation (How do they define evaluation? Does 
this match your definition?) 

• methodology (Do they suggest a technique which is realistic 
within the limits of your project? Do you understand all the 
implications of this technique?) * 

It. is important in securing the services of " external evaluators to 
examine their evaluations of other programs,. to understand the technique 
and philosophic approach of the evaluator. Calls to past clients are a* 
good r idea. * ~ — - 1 

Who will be responsible ? 

for the design of the evaluation [ .? 

•for the collection of information ? 

for \ the* development of instruments \ ? 

• for the 3 * analysis ;j , ? 

/» — " : : 

Responsibilities should be clearly delineated and should be divided 
among the persons involved v using the above tasks as a guide. 

What are the cost factors involved in evaluation, in terms of money, * 
time, and resources ^ 1 ~ 9 I TT \ 
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Evaluation should be considered a cost-efficient measure* not a costly 
one. Things to be considered are: 

• • What resources do you have available? , 

• Do you "have an evaluation budget? (A good ri*fe of thumb here is 
not to exceed more than ten percent of the tQtal project budget)' 

• Have you accounted for program time needed to. respond to 
evaluation requests? (Even if you employ an external evaluator, 
staff time will be expended.) ' ' 

• Have yoji measured the potential- impact of the evaluation against 
the anticipated expenditure? (Sometimes expending money at the 
beginning of the project will result in greater savings later in 
the pro4ect life.) ' v 

. • What are the "costly" considerations, such as invasion of 
privacy, defensive attitudes of staff and clients, and 
difficulties encountered in information collection. 
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FIVE COMMANDMENTS FROM tl/.H. AUPEN* 

Thou shait not ans^fer questionnaires, - 
or quizafes upon world affairs, nor. 
withjiompliance take any * test* Thou 
sJpdXt not sit; with statisticians, nor 
" / £6inmit a social science* 

*F^om "Undvi thz Lukz - A RzvotutconaAjj 



An evaluation strategy is simply the w'ay in which one goes about 
analyzing evaluation information. All evaluation' strategies, or models, 
can utilize almost any type of available data. They distinguish 
themselves in terms of the questions they pose, how information is 
arrayed, and how decisions will be formulated. What follows is an . 
overview of seven of t hi major evaluation models, their features , uses , 
and procedures . It sjKjftld be noted that these evaluation models are only 
presented as sample techniques^ they need not be used exactly as' 1 - 
described, nor are they "required" of Project Directors. Many successful 
evaluations have been done merging oneor two of the techniques, or using 
features from several of them. Trie important point here is to select a 9 - 
strategy which you 'feel comfortable with, and wlrich will be most 
satisfactory to your audience. ' r 

Discrepancy Evaluation: The Model of Malco\m Provus 
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To use this me,del f , you compare the 
differences between standards you've set for 
performance, and %he actual levels of 
performance: the outcomes of your project 
are compared wtth your objectives within the 
context of program operatiofu 

In order to do this, you have to be able to 
clarify your goals, not just in terms of end 
products, but the goa.ls you set for the • 
design, operation, and production of interim and final products, and far 
costs. 

' * 

At all of these stages* indicators* of performance' are compared with 
standards which Serve as criteria, of performance, and discrepancies 
between performance and standards are reported to program managers* A 
discrepancy is defined as a simple difference and need not be negative - 
i.e., it is a discrepancy to be operating under budget yet certainly not 
negative* ' . ' 
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- . • f hen you adm1n1 ster a program, you keep in mind your goals, or plansi 
.* . and your actual attainments, or results, but when ypu use the discrepancy 
model, for evaluating, you concentrate on the difference. between goals and 

results which is the "process"* of your program. 

• - • . " ' ' «\ " 

What's found by studying the 'differences between goals and results is 
^ • a -set of "tearnings'\ which Tcan ihflaence the results of ypur program, if* 

you use them. * ■ • . j , 

Oftentimes, the^ Discrepancy Model is used like a checklist or report 
'card, tha* is: this is what you said you were going to do, now let's see 
whether or not you did it. But the -model's intent is far we than 

• • • - this. It can aid you -in. doing what you sajd yoo were going to do and 
J .further to help yeu determine whether on northis is still" a desirable 

goal.. - ' 

/. Provus advocated continuous cojrftufiicatjon between project staff and- 
evaluation staff, so that discrepancy information be used at the 
j earliest possible occasion to.modify the program whenever managers feel 

. modification is needed to attain the desired outcomes. In the 4 Provus 
, model,.,.1t , s important to remember that for purposes of evaluation; V ' 
discrepancies are not necessarily negative-differences need not be 
deficits, as they help you get a 'sense of what you're doing. According- 
. to this mode], the place where you end. up is affected not only by where 
: you are,- but also by what's, happened along the way,' jf you keep an >eye on 
the differences between where you are and where you thought you'd be. 

'* The ' key features of -the Discrepancy Model are: " 

t the need for a close link between program planning and evaluation 
' j| c . f t evaluation is a continuous -process *•« 

t evaluation criteria need to be linked to stages of program 
• " development and implementation 

1 ' ' * evaluati tSCrePa " Cy infbrma - t1on as tne Primary method of program- 

K ' • the need to maintain rapport between evaluators and proqram 

> managers. - . r 9 
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" -Goal-Free or Consumer-Oriented Evaluation : the model of Michael 
5criven '< 

Goal-Free Evaluation, though the title of- 
this model conjures., up a variety .of images,* 
does, not imply that the evaluation, .project 
or evaluator is without goals. Rather thjs 
model ts concerned with measuring a]2 of the 
effects of a project and Scriven contends 
that this can only be done-if the project"""!? 
viewed holistically apart from its original 
goals or intents. / 

The model is based on- the assumption that 
consumers don't usually care about whether a program achieves /its 
organizational goals— they care about what a program really d/bes, whether 
or not it!s a "good" program, and they expect an evaluation to deliver* 
this information to them. Scriven feels that these consumer "values" are 
embodied in standards which support the design, implementation and 
evaluation of a program. If evaluators are in fact saddled with this 
responsibility, they need to put a lot of- emphasis into understanding the 
standards they- use for evaluation: are they subjective, biased, 
- appropriate? Scriven feel s^ that the program professional (e.g., the arts 
^educator) is best suited to conduct the evaluation as these persons are 
most. familiar with the standards to be used. 

One of the features of Scriven \s model is that it can use comparison, 
either between programs, or in the same program "before and after," or in 
the field at large. A non-comparative technique also can be used. 
, What's important is to determine whether or not the audiences of the 
evaluation will be making comparisons and to- gather information * 
accordingly.- 

In developing his evaluation model, Scriven distinguishes between 
"formative evaluation" (used to improve a program while its being 
implemented) and "summatfve evaluation" (used to judge- the completed 
program, process, or product). Formative evaluation should be done by '' 
someone "inhouse," while summative evaluation is done by an independent, 
outside evaluator. 
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Scriven also distinguishes between "intrinsic evaluation" which 
examines the '"means" ,* and "pay-off" evaluation which examines the •'ends." 

If 

Scriven developed a taxonomy of criteria for education evaluation 
studies which can be used as a checklist in designing an evaluation. The 
steps which Scriven developed for designing an evaluation methodology are: 

(1) Defining the type of evaluation needed by intended use— forma- 
tive, summative or both ' 

• (2) Defining the role of the eval.uator(s) 

(3) Determining whether the goals of the program are adequate, and 
worthwhile 

<(4) Determining the instruments' to be evaluated (intrinsic evalua- 
tion) / 

(5) Determining the effects of the instrument(s) (pay-off evaluation) 

(6) Determining whether the evaluation is to.be comparative or non- . 
• comparative. 

Scriven's methodology for evaluation is more appropriately thought/ of as 
a set of conceptual distinctions (and thus related to evaluation design) 
than a procedural model for conducting an evaluation. 

/ 

t- ^ 
* •* < i " 

i ^ * 

Illuminative Evaluation : The model of Parlett and Hamilton 

The focus of this evaluation strategy is on 
describing and interpreting, rather than on 
measuring and predicting. Your goal as 
evaluator is^to obtain the best picture of 
what's going on with the project— you use 
descriptive data obtained from observations, 
interviews, questionnaires, and .archival 
records. After you 1 ve' described the [ 
project, you sfiare this information with 
your staff, clients, and others who are 
interested and/or involved in the project, the purpose being to assure 
that there is consensus on what the activities of the program were. 
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The next step in this process is to interpret the program according 
to the interests and needs of the staff, clients and other. interested 
groups. This step is made in order' to gain some understanding of the ■ 
project's diversity as' perceived by external audiences. As evaluator, 
using this model, you~want to know what 2s, and whyj describe and 
interpret! The entire program should be studied in depth— its rationale, 
evolution, operations, achievements, and problems— in other words, the 
, program in relationship to the environment in which it operates. 

J 

This approach i.s intended to ,be unstructured. The assumption behind 
it is that pre-defined hypotheses, which you want to test, end up 
constraining your data gathering, and unduly influence the nature of the 
findings. •'• • « 

The developers of this model specified three stages for the evalua- 
tion: - 

• observation 

• further inquiry ' . ' 4<s 

• explanation ••-,<•.♦ , 

Each of these stages is "conducted independently— the second and third 
stage building on the first stage. Observation is directed towards 
program activities. The second stage or. further inquiry is used to 
provide information which will "i Illuminate" the program activities in 
terms of goals and outcomes. * The final stage is used to explain or 
analyze the program in terms of goals, activities and outcomes. 

- Responsive Evaluation : the jnode l of Robert Stake 

'~— — S 3 1 

This jjiodel focuses* on both description and 
judgment as being necessary parts of « , 
•evaluation* In malcing a judgment, you are 
assessing the worth of a particular aspect 
of your program,- e.g., an educational 
strategy, as it relates to improving an 
aspect of learning. What you're trying to 
do is make a judgment about your program, 
and share it with an external audience. 
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An important aspect of Stake's Model is the distinction of data 

sources by their relationship. in,time to the practice being evaluated: 

- | * ' 

o Antecedents— those conditions existing prior to the teaching and 
learning experiences which may relate to outcomes, such as the 
students' aptitude^ or previous experiences, previous 
curriculum, etc; I - 

o Transactions— interactions of persorjs "in the learning 
environment, your program. 

o » Outcomes— the results of antecedents and transactions. 

In this model you look at things from two perspectives— the . 
antecedents, transactions and outcomes which you intended , and those 
which. you observe . In evaluation, you are looking for congruence between 
what you intended and what you observe. This also combines the use of 
observaTions to obtain descriptive information. But it is more 
structured than other models in that the program intents, or goals, serve 
as the framework, for gathering information. And the emphasis- in this 
model on the relationships (or contingencies, as Stake refers to th 
between the antecedents, transactions, and outcomes mak€s it more 
environmental ly-respcinsive than the Provus Discrepancy Model • 

In essence, using this model you are making judgments about your 
program all along the way—are things good or bad— and you're . 
continuously checking out your assumptions; 

In this model, the descriptive data that you obtain Sire based on 
logic;- while the judgments are based on empirical comparisons between 
results (outcomes) and standards, THe model looks at logical 
consistencies in order to understand diversity between program intents 
dnd program operations . , ^ ,r 

The basic procedure of this model is to make a determinati.orv as to 
what decisions need to be made and then to gather information to inform 
these decisions and/or Recommend change strategies* 
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CIPP (Context, Input. Process', Product) Evaluation, or 
Dec1s1on-Or lented Evaluation: the model of Daniel Stu 
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The CIPP model, a combination of other 
evaluation models, was designed to remedy* 
problems In previous models. .It assumes - 
that there will- be a largetnutual influence 
between evaluation and ^ecision-maklng. 

The thfee major steps in the evaluation 
process are delineating, obtaining, and 
providing information for judging decision 
alternatives: 



t 



delineating information refers to identifying required \ 
information 

obtaining information includes collecting and analyzing the data 

providing information refers to the organization of information 
for reporting purposes. 



Seen from the perspective of this model, there are three environ- 
ments, or contexts, in which changes can occur, within which/^efcisions by 
managers must be made: 



those guided by technical standards, with a routine, cyclical 
data collection system 

those in which there is ongoing developmental activity aimed at 
continuous improvement " . J- 

those in which innovative activities are being used for 
inventing, testing, and diffusing new solutions to problems. „ 

fa this model four types of decisions are delirj^ated^and determine- 
the type of evaluation to be conducted: 



planning decisions : these are aimed at major program changes, 
based pn discrepancy information*. The evalu^tor performs a • 
" context evaluation" in which the relevant environment ts 
defined; intended and actual outcomes and discrepancies between 
them are described; problems tHat prevent a high degree of 
congruence are diagnosed. • v . ) 

structuring decisions : these specify the means to dnds that 
were established through ^planning decisions. The kilnds of 
variables studied include program organization, personnel 
schedules, etc. " Input evaluation" is used to provide 
information on methods of resource utilization for meeting \ 
program goals. [ 
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o ' Implementing decisions : these involve carrying out the planned 
actions. " Process evaluation" is directed towards these 
decisions needs in several ways— detecting or predicting 
problems in the procedural design or its implementation; ^ 
providing information for programming decisions; and maintaining 
a record of implementation processes as they occur. . 

o Recycling, decisions : -those which determine the course of any 
activity at any given point in the program. " Product 
evaluation" serves these decision needs by measuring and 
interpreting attainments, both during and after a project." 

Art-Criticism Evaluation : the model of Elliott Eisner 
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This evaluation model is patterned after the 
/ process of art criticism. It's also called 
* "connoisseur-based evaluation." 

Programs ajpe^evaluated either by expert, 
so-called "blue-ribbon," panels; or they are 
compared to recognized standards of 
excellence. These standards of excellence 
■"need not be only those from the. field at 
large, but more importantly, these standards 
should be recognized by -the audiences of the 
specific project. Standards can be as measurement-oriented as increases 
in test scores or qualitatively-oriented such as artistic excellence. A 
blue-ribbon panel can be composed of recognized experts in the field or' 
professionals in the field <ran assume the role of the panel by using 
agreed-upon standard? for/judgment, e.g. j a group of music teachers - 
within a school arts? probram can be ( , brought together and establish 
cri tenia' for judging; their own program. _ 

The goal is to critically describe, appraise, or illuminate a program 
element, and to determine what merits or demerits distinguish a program' 
from others that are at the same level of development. 

Basically, the goal is to pjade a "stamp of approval" on a program by 
determining "would a critic approve this program?" 
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As a model, it is particularly BeWident on the^evaluators' expertise 
and sensitivities, and on their ability T^onvey meaning and feelings. 
This model assumes that judgment is based on affective information or 

"feelings" and so evaluations' shoulci address the affective domain. 

< - 
Another feature of this model is that by utilizing critical' standards 

or criteria as a basis for judgment, you not on ly, determine the project's 

status in relationship to these stSi^dards, but you\ can seek to improve or 

"raise" the standards of the field. ) 

Jurisprudence or Adversary Evaluation ; the model of Robert Wolf 

The jurisprudence/ adversarial evaluation 
model is characterized by: information 
management by courtroom procedures; use of ' 
citizens for information review; and bipolar 
exploration of key issues. Two evaluation 
. tebms are organized. One team serves as the 
Advocate team and seeks to gather 
information and develop logical arguments in 
support 'of the program being evaluated. A 
second team serves as the adversary team and 
attempts to present information highlighting 
the inadequacies and weak-points of the program. Neither of these 
evaluation teams is responsible for the ultimate evaluation of the 
program; rather, their respective responsibilities are to collect # 
relevant information and to develop evidence and arguments in support of' 
their position for or against the program. 

A jury of "citizen evaluators" hears the evidence andarguments of 
each evaluation team. The evidence is presented in court room styje with 
various witnesses giving testimony (a review of test scores, explanation 
of decisions, descriptions" of the effects of certain 'decisions*, etc.). 

9 
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Each team is allowed the opportunity to "cross-examine" witnesses. 
This permits the development and exploration of alternative hypotheses. 
The 5 jury of citizen eval tutors may direct questions (funneled through the 
jury foreman) to any witness for the purposes of clarification or to 
examine other hypotheses. 

After both evaluation teams have completed their presentation of. 
evidence and arguments and after each team has presented summary 
statements, the jury of citizen evaluators retires to deliberate on the 
evidence that has been presented. It is then the responsibility of this 
citizen jury to make judgments concerning the program being evaluated 
point- by-point for each of the prespecified key issues. The jury may 
report favorably or unfavorably, in part or whole, and may also identify 
areas where the evidence is too. contradictory, too minimal, or too . 
one-sided and thus withhold judpent on certain issues. 

The jurisprudence model has three advantages not offered by any other 
evaluation strategy: 1) it insures the full exploration of key issues 
through the presentation of evidence and arguments reflecting! both sides 
of the issue; 2) it enhances the opportunity, to develop alternative 
inferences from data; and 3) it removes from the evaluator, the %4 
responsibility for summary judgments and placfes this responsibility with 
a panel of citizen jurists. r. 

The most obvious disadvantages are increased cost (by funding two 
evaluation teams rather than one) and the administrative complexity of 
agreeing on evidentiary rules, specifying the- scope of key 'issues to be 
addressed, and selecting instrumentation common to both evaluation 
teams. Some of these "disadvantages" can be countered by utilizing 
advisory panel members".as the evaluation team. 
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V. SUMMARY 

*f, f 

The documentation, evaluation. and dissemination strategies in this 
guidebook have been presented Within a larger context which views the 
management of information as a process as important to the success of a 
program as that of managing resources and personnel. 

Moreover, these ideas- have been based upon our particular view of .the* 
needs and uses for information in Federal demonstration programs such as 
the USOE Arts -Education Program. .Within this perspective national, 
state, local, and individual information needs are considered 
interdependent . Thus, effective information management starts with the 
experiences of the local program staff and clients and, to the extent it 
provides useful information at each \ level of program support, it 



eventually produces the quality of information needed for national 
management and dissemination needs. \ 

- i i ( \ » a 

Finally, we view information as more than experience- derived data. 

It also can be used as part of experiences, and especially to influence 

others. An Arts Education pro4ect does have a complex task: as a 

"temporary system" it must have a positive influence on individuals and 

permanent, social institutions over which it has little or no direct 

control , v Each of these has its own goal\s, objectives and primary 

responsibilities. It is for this reason! that documentation, evaluation 

and dissemination can be major management! strategies. Many of the 

purposes of an Arts Education program are\determined and shared by the 

individuals and institutions with which it\ works. To the extent, that 

these individuals and institutions are provided with -information to 

perceive the achievement of their purposes through the project, tp that* 

extent the Arts Education project may successfuly influence changes in 
c - \ i 

the permanent institutions with which it interacts. In other words, when 

individuals perceive thejf purposes and growth enhanced by an Arts 

Education project's activities, the probability of achieving -project 

- objectives increases and persons Both within the project and representing 

cooperating institutions feel ownership in the project and its strategies . 

66 
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Effective management of information in Arts Education project 
management, to usi therefore, provides: ' 

• - a unifying process which links the niseds of the individuals and 

- institutions; 

• a learning process which enables a project to stay on top of the 
complex and changing conditions it faces by accurately 
perceiving its environment,* drawing information from situations 

* while still in process, learniny from these situations, and 
applying, the learning to its advantage;', 

• an involving .'process which .recognizes that while management ! 
decisions may be made by one person, no single individual has* 
the experience or information to support the .decision. When all 
project members share' information, share in decision-making, and 

* understand clearly the rules, Regulations, and operating limits, 
the project runs well and pai^ons gain psychic rewards from 
participation; " c 

% a controllable process which acknowledges that, while the 
functioning of many of the project's cdmponents are only 
indirectly controlled by the project-, "management 6f information 
/ 0 is one major function which is totally under the project's 

control. Too often a project will relinquish what is actually 
its "power" to affect its own outcomes because it feels that 
documentation, evaluation or dissemination are functions that 
have to be done for Washington, or that have to be 'separate 
components distinct 'from other "objectives. In these'cases, 
management of information often becomes* a purpose in itself for 
which onjy a few people are responsible, and in whose successful 

accomplishment there is 'little investment by others. 

• i 

«• and finally, information management is a natural process— ah 

ongoinj problem-solving process that responds to the basic human 
need to have some influence on surrounding* conditions. The 
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individuals to be 'involved in the pro, 



j ect-*- admi n i str ator s , 



parents, support personnel— already have their personal 



"management" system in plaqe ftejrmitti 
to select appropriate actions and to 
again* The major management task for 



director is one of providing these individuals and groups with 



the information and opportunities to 



problem-solving abilities and interest to the accomplishment of 
project objectives . . . 



ig them to identify- needs, 
self -correct before acting 
the Arts Education project 



groups 

apply their natura l" 



^..communication requires shared 
, experience. • 

Vitvuf. Vmckvt 
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Two final thoughts V . • 

about information that goes jr^, and corfies out , 

of our programs . . . 
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'•The government arc very keen on 
amassing statistics. They collect them, 
add them, raise them to the «th power, 
take the cube root and prepare wonder- 
ful* diagrams. But you. must never 
forget that every one of these figures 
comes "in the first instance fltom the\ 
village watchman, who just puts 
down what he damn pleases 

Sir rfsiah Stamp 

INLAND REVENUE DEPARTMENT (Eofbad) it&lgig 
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THE, GOVERNMENT IS VERY KEEN ON PRODUCING . 
EXEMPLARY PRODUCTS AND PROGRAM MODEL'S. THEY RESEARCH 
THE NEED FOR THEM, DEVELOP, IMPLEMENT AND EVALUATE 
THEM; DISSEMINATE AND DIFFUSE THEM; AND PRODUCE 
WONDERFUL REPORTS ABOUT THEM! BUT YOU MUST NEVER 
FORGET- THAT EVERY ONE OF THESE. PROGRAMS OR PRODUCTS 
' WILL, IN THECAL INSTANCE, JBE USED BY A HUMAN 
BEING WHO WILL DO WHATEVER HE DAMN PLEASES TO MEET 
tHS NEEDS FOR SATISFACTION IN\HIS WORK, 

, Lewis A. Rhodes 
' (USA) 1976 r 
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WHERE WE STARTED . . , 



This current effort of the USOE Arts and Humanities staff to pro- 
vide assistance in evaluation, documentation and dissemination is 
limited to a 2-day workshop. However, it is- built upon ah understand- 
ing t-kat has been developing over the past few years concerning the 
unique needs of the art educator who often functions as a program 
-manager 'in a local or state educational system. 

there have been several state and national meetings a-nd con- 
fereaces to explore the particular eva^u^Jfedo'n concerns of the art' edu- 
cator. We have attempted to utilize the finding's^ of these meetings as 
\ ■ indicators of the directions this workshop should go. For example, 
a an March 1977 the Department of Public In§jtruction of North Carolina 
■uid a symposium on "Measurement in the Arts: The Search for X. 11 
^ ^ii. ivnpciium was held to give participants a view of the many efforts 
• being* made to assess arts programs and their e'ffects on studefnts. 

There were several conclusions that were felt ^o be of major import- « 
ance. - . 

/^^\ m "1. We need to give much more thought to the. reasons we . . 

r]y * ^ collect data., 

2. We need to give much more thought to/the use we plan 
4 to make of data. 

3. We ileed to make far greater use of the immense amount 
v ^of data n.owpat ha^d. 

4. We need tojU^ok more carefully at the consequences of - 
'xlata collection as those consequences relate to our 

bestdefi^edyde^sires for children. since -the simple 
acrtTof coTVerting data lets loosTe implications over 
which the collectors have no control. 11 , 

In March 1978, the American- Theatre Association and USOE con- 
ducted a seminar entitled "The Analysis and Dissemination of Arts m 
Education Evaluation -Techniques." The concept of the seminar cen- 
tered on the fact that * 

. the demand for evaluation of arts^ proj ects has grown in 
the past ten years. Since arts evaluation is still in its 
infancy, few state and regional arts practitioners -and J 
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' leaders know very much about its function and methods. 

.What is appropriate, useful, manageable and cost-effective 
for arts-in-education projects is critically * important to" 
* the growth and development - of new programs. Therefore, 
there is a need for: 1) knowledge of methods available 
for evaluation. of arts projects, and 2) dissemination of 
the information to project administrators and directors in 
the field.- o ■ ; 

One of the seminar participants, Dr. Jerome Hausman, presented 
several comments and recommendations at the conclusion of the seminar 
He felt that k 

...within each of the arts fields, there are 'problems and 
ambiguities pertaining to evaluation. * This is "built-in" 
v by virtue of the dynamics of the arts themselves — changing 
conceptions for form, content*, and context. All of this 
is further compounded by thinking that links the arts and • 
environment. No wonder we are loathe to develop simplis- 
tic approaches to evaluating "state-wide" arts in educa-: > 
tion efforts. No wonder there is a turning toward more 
generalized- data sources and more open-ended and flexible 
evaluation strategies. We have .found that overall evalua- 
tion strategies need to be- responsive to multiple and per- 
haps diverse criteria. But, we need to be clearer and more 
structured in the strategies we develop. 

My own tendency toward self -study and anecdotal review 
. procedures is based upon the assumption that critical 
issues must.be reviewed^ in context. We also need to 
develop means to' distill .critical elements for analysis 
and assessment. Bob \£take f s use of terms like portrayal 
and depiction are very usefui and exciting to me. The 
real challenge rests. in telling the story with attention 
to critical details and then assessing the meaning and 
significance of daja. Of course, measurement approaches 
and techniques may be applicable along with assessment 
of performance or /outcomes. 

HauSman goes on' to point out that 

•...evaluation strategies need to^address themselves to 
processes and outcomes accountable to the arts! Once we 
have developed our general picture, specific criteria 
pertaining tor behaviors, skills, and outcomes should be 
distilled from this general picture. Responsibilities 
for persuasion should only be assumed^ after the data are 
reconciled with values and purposes consistent with the 
program's objectives. ^ 
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Ijj addition to these meetings, the writings of both arts educa- 
, tion and evaluation theorists reflect the concerns for evaluation* 
Robert Stake finds that * 

• . .accredition reports rarely show the real worth of an 
educational program. ■ 4 

Michael Scriven says that * % 

...if the goal of evaluation is to determine wortlr, ' then, 
to collect data, focus inquiry and inspect causal rela- 
tionships .without making a judgment .of woHh is not 
evaluation. * 

0 

S 

Stake elaborates on this idea by stating that , controls , tests, regres 
sion equations and behavioral language may all be essential for test 
construction but may not facilitate the conduct of an^ evaluation proj 
-ect at all. These theorists have responded to people who are con- 
<- 'Vied with documentation, evaluation and dissemination in the field 
c ~ a:*s education. Laura Chapman writes that * 

...the purpose of evaluation is to discover the value or 
significance of something. Measurement is not an integral 
part of evaluation; it imposes on the concept of "worth" 
an ordering and ranking system which leads to a denial of 
intrinsic value. An evaluation of an arts program should 
be empirical, valid, broadly reliable and responsive to 
the qualitative character of art experience. 

Stake takes this theme even further when lie says f ^ . 

./.We need a reporting procedure £fj>r facilitating 
vicarious .experience. We need to portray complexity, x 
holistic impression, mood and even the mystery of -exp<pi> 
ence. < 

Finally, j^e have also based this workshop on the comments which 
were expressed by current Arts and Humanities grant, project directors 

• "Evaluation and research are essentially .the same thing." 

• "Documentation and dissemination are devices to show 
others what we did that was good'/" 

• . "Sophistication in evaluation means a strong preordinate 
. design usually done by someone else." 
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"If we do deal with the relationship between what is 
actually done and what is evaluated, might not Arts 
Education lose out." 

"Evaluation means numbers to jsftbw administrators • " 

"Evaluation is never simple, if it is, it is not good 
enough. Our teacher-made tests are just good for us 
but they are not what a real evaluator would do." 

"Good evaluations are expensive and take a lot\ af money 
and time to do." * 

& • * 
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SELECTED TESTS IN THE ARTS 

Creativity 

Torrance, E.P. 

ASX AND GUESS TEST 

Neutral forms, age, originality, fluency, flexibility, el aborati 
production (written) 
'Personnel Press, Inc. 

Item: Consist of three activities: 

I. AsMng - subject required to 11st as many questions as he can 
% about a drawing presented to him. • f 

II. Guessing Causes - subject required to 11st as many as possible 
causes of the action 1n the given drawing. 

III. Guessing Consequences - subject required to 11st as many 
possibilities as he can of what m1gh"t happen as a result of what' 
Is taking, place 1n the picture. 

Scoring: Bas«d on Fluency (no. of relevant responses), Flexibility 
(1 point given for utilization of each of several categories given 
1n the scoring manual), and originality of responses. 




Validity: The author presents considerable detailed Information 
• on content validity, construd>valid1ty, concurrent validity, 
scoring reffabimyr a/id test-retest reliability 1n the technical 
manual of the test series. 

Eisner, E.W.» 

BUILDING TEST 

Art, unspecified, originality, production (building, drawing) 
Typology tff creat1v1ty n 1n. the Visual Arts. Eisner, E.W. 
Studies in Art Ed., 19&, 4(1)-, 11-22 

* « 

Item: Subject must construct an object out of 1/4 pound of 
clay and xolored tooth picks. 

Scoring: Based on opinions of 3 trained judges. Evaluation 
proceeded on .a nine-point scale and objects scored- orfse" 
criteria: Boundary Pushing (subject and form), Boundary 
(subject and form) aesthetic organizing. 

Torrance, E.P. . . , .. ' 

CIRCLES TEST 

s 

# Neutral forms^ age, originality, fluency. flexibility, elaboration, 
production (building, drawing) . 
Personnel Press, Inc., Princeton, N.J. 

\ ' .-'^ / ' •/ 

( Item: Subject presented with a series of 36 circles printed on 
2 pages and 1s Instructed to> draw as many different objects as 
he .can Incorporate the circles. . • . .» 
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Item: Four Items-Forms i Subject lists different things each of 5 
ambiguous silhouettes cpuld possibly be. 
Functions: Subject lists different uses for each of 4 common 
objects. / 

Concepts: Given an attribute of a class of objects, e.g. "things 

> round", subject must list appropriate objects. 

i 

Scoring: Scoring for each test 'based on frequency (no. of responses) 
variety (no. of different classes of responses ) fx 
and uniqueness (ia frequency , of occurence). 



Validity: The author? presents detailed information regarding 
validity and reliability in the article. 

Torrance, E.P. / 
PICTURE COMPLETION ]EST 

j 

Neutral forms, age,/ originality, fluency, flexibility, 
elaboration, production (building, drawing) 
Personnel Press, Inc. Princeton, N.J. 

^Uem: Subject given a series of 10 drawings of incomplete 
frfgures* is instructed to complete a drawing using the 
'lines 0 given^^ / — . 

Scorinq: Based qji fluency (No. of pictures completed), 
fl exi bffity (No J of different categories of responses), 
originality (Imagination), and elaboration (Amount of detail). 

Validity: The author presents considerable detailed information 
an content validity, construct validity, concurrent validity, 
✓ scoring, and te^t-retest reliability in the -technical manual 
bf\the test seryies. " * J 

Torrance, E.Pi / 

PICTURE CONSTRUCTION (CURVED SHAPE JJTEST 

Neutral fonn^tage, originality, elaboration, reproduction 
(building, dravjRn^ J 
Personnel Press Vine, Princeton, N.J. 
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Item: Subject gtven a piece of colored paper* in the form of 
a-curved shape and-must draw a picture utilizing it. 

Scoring: Based on originality (on a. scale from zero to five), 
and elaboration ^amount of" detail). . 

t « * 

Validity:. /The author presents considerable detailed information 
on content/validity, construct validity, concurrent validity, 
scoring? and test-retest reliability in the technical manual 
of the seryies. 
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Likert, R. Quasha, W. H. ' 
MINNESOTA PAPER FORM BOARD TEST 

Form, age perceptual" grouping, successive categories 
Psychological Corporation, 304 E. 45th Street 
New York', New York 10017 

Item: 64 multiple choice items each consisting of a geometric 
figure divided into 2 or. more parts. The subject must choose 
one of five figures presented that would represent the 
original should the parts be fitted together. 

Scoring: The authors present considerable data on norms, 
validity, and reliability in the manual accompanying the 
test. 
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C.-.nd, Irwin L. 

°R£rERENCE AND AESTHETIC SENSITIVITY TEST 



( 



Painting, age, preference, rank order 
- (t'crsonel preferences as an expression of- aesthetic sensitivity 
•}> Irwin L. Child, Journal of Personality, 1962 vol . -30 pp 496- 

512 

Test consists of 12 sets of pictures each set containing 60 
pictures. Subject must rate reproductions of paintings 
according to his preferences by 'dividing each set of pictures 
into, 1.0 piles of ,6 pictures each and rating each group of 6 
individually. 

■ Scoring: Norms derived by ,the scaling of reproductions by 
13" art graduate students according to aesthetic quality. 

Reliability: Determined by test-retest method yielding a 
correlation of approx. .90 for 44 undergraduates. 

Benner, G. Seashore, H. Wesman, A.G. 

SPACE RELATIONS - DIFFERENTIAL APTITUDE' TEST BATTERY 

Form,\age, perceptual grouping, successive categories 
Psychological Corporation 304 E. 45th Street ' 
New Yorki'N.Y. . * * • 

Grades 8 thru 12 * ', 

t . 

~ Item: Test consists of 40 patterns which can be folded into 
figures. For each pattern, five figures are shown. The * 
subjects must decide which of the figures can be made from 
the patterru 

Scoring: Extensive information on scoring and norms Is 
.presented in the manual of the test series. 

> • \ - 

Lewerehz* Alfred -S. - 

TESTS IN FUNDAMENTAL ABILITIES OF VJSUAL ARTS' v 

California Test Bureau Delrfronte Research Park 
Monterey, Calif. . j 

7 . - 7S ' 
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Eisner, Ellfot W. . ' - 

' EISNER ART INFORMATION INVENTORY 

4 

Questionnaire, age, background knowledge, understanding, 
standard objective - • »• - 

Arts Curricula for the gifted. 
5 . Eisner, Elliot W. Teachers College Record, 1966, 67(7) 
492-501 * 

Consists of 60 multiple choice Items equally divided Jnto' 
subtests dealing with art terms, art media and processes, 
artists and their works, and art history. 

Scoring: No norms given. 

Reliability: Application of the Kuder-R1chardson Formula w 
20 to the tests results of 1488 subjects yielded a coefficient 
of. 934. 

Graves, Maltland 4 s 
GRAVES DESIGN JUDGEMENT TEST 

Figure, age, sculpture, preference, pair comparisons 
Psychological Corporation, 304 East 45th Street ~ 
New York 17, N.Y. 

Item: Ninety Items each consisting of 2 or 3 plates from" * 
which the subject must select the one he considers best. 

Scoring: Scoring key furnished. Nor&is given in percentiles 
.for high school and college art and. non-art students. ' — 

Norms- derived from test results of 826 art students and 480 

non?art students. • 

» . « - 

Reliability: Coefficients range from to .93 with a 
median of .86 based on retests of 14 groups of students * » 
from 2 high schools and 3 post high school, Institutions. 

Horn, Charles,* C. . , . 
HORN ART APTITUDE INVENTORY 

Task, age, drawing, production / 
. C.H. Stoeltlng Co. . ' ' 1 %. 

424 North Homan Avenue * * 

Chicago 24, 111. 

Item: Part One (scribbling and doodling): Subject must sketch 
• twenty' familiar objects. . , 
Part Two (Imagery), subject must manipulate twelve rectangles 
conta1n1ng/-"key" lines to arrive at a picture. 

* 4 

Scoring: Items rated as either excellent, average, or poor 
ba,sed on examples of the 3 levels of work Illustrated 1n the 
manual? No method given for. obtaining quantitative scores. 

i .• I nSf^i 

Validity:, Determined* by corrections between, two scores and 
teacher ratings of 36 high schpjpl senjors, the product-moment . 
correlation coefficient equaled .659/ * . f • .t 
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LECTED TESTS IN THE AR^S 



Dance 



McCulloch, Margaret L. 

* TEST OF RHYTHMIC RESPONSE 

Aural (music), age, rhythmic performance, motor performance. 
The development of a test of rhythmic response through, 
movement of first grade children. McCullpch, Margaret L. 
Thesis (Ed.D.) Univ. of Oregon. t 

Item: Test consists of 14 items requiring the subject to 
step, clap, walk, and bend in time to given rhythmic patterns. 
- Test purports to measure response to pulse beat, response to 
accent, response to rhythmic patterns, and response to musical 
phrasing. 

Scoring: Based on a rating scale^from 1 to 3 indicating 
whether the subject was always, sometimes, or never in time 
with the music. 

Validity: Test items selected by a dance and a music "specialist", 

'Reliability: Test-retest reliability coefficient reported to 
* •. • be .90 based on^test results of 27 first-grade children. 

Coppock,. D.E. . 
tfEST OF RHYTHMIC MOTOR RESPONSE 

£ 

Aural (music), rhythmic* jJerformance, motor performance^ 
Development of an objective measure of rhythmic motor response. 
Copppck, D.E. Doctoral Dissertation, Urriv. of Iowa, 1964. ' 9 

<r m Item: Subject is presented- with 23 rhythmic patterns (drum) 

of increasing complexity*ahd is* required to step in time to 

each pattern. 

\ ^^^^^^ 

* Scoring: Based on a 5 point- rating scale (from 0 to 4). 

Subjects rated on meter, tempo, and total for eacftitem. 

Validity: Determined bjj cancelations between test results and 
teacher ratings of 92 subjects. Coefficient found to be .267 
. * (p. 01) A 

Reliability: Based on test results of 92 subjects. Spearman- 
Brown Prophecy Formula yielded a coefficient of .91. 

Withers; Maida Rust 

• CREATIVITY OF MODERN DANCERS 

Action, dance training, dance perf orma'nce , motor performance. 
"Measuring the Creativity of Modern Dancers," by.Majda Rust 
Withers. Masters Thesis University of Utah, 1960 V . 

ERJC - 9 80 . . x ; 
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Music 



, SELECTED TESTS IN T-IE ARTS 



Al1fer1s, James Steckl§1n, John E, .' 

AUFERIS-STECKLEIN MUSIC ACHIEVEMENT TES 

Rhythm, melody, hartnony, age, aural 1den 
reading, standard objective v 
University of Minnesota Pres's Minneajaoli 



Item: Consists of 79 items at a piano miksi 
compared with printed notations of melody 
Multiple choice of 4 alternatives per pl<yed 



Scoring: Norms given are based on the 
students. .Separate norms, for geographicel 
instrument studied, arid type of school 



Beach, Frank A. Schramnel , H.E. 
BEACH MUSIC "TEST 



Music, pitch, musical training, age," aural 
composer identification, visual note reading, pair comparisons, 
standard objective 

Bureau 'of Educational Measurements Kansas 
Emporia, Kansas 



ification, visual note. 
14, Minn. - 



c (played) to be 
, rhythm, or harmony, 
item. 



testing of 2500 music 
location, type of 



1 identification, ' 



State Teachers College 



Item: 70 items divided into eleven sub-tosts consistinq of 



knowledge of mu^jcal symbols, recognition of measure, tone 
direction and similarity, pitch d1scr1m1ncit1on, application of 
syllables, time value, musical terms" and symbols, correction 
of notation, syllables -and pitch names, representation of 
■ " pitches, composers and artists. » • 

Scoring: Norms given 1n percentiles for grades' 7 through college. 

Validity: Based on test scores and teacher rankings of several 
thousand students. Correlations with mus1:1ansnip = .65, with 
. knowledge of musical fundamentals = .74. 

.Drake, Raleigh M. 

DRAKE MUSICAL APTITUDE TESTS 



81 



1on playing, visual 



Rhythm, 'melody, verbalization, age, percuss 
-note reading, pair comparisons, 
Science Research- Associates, Inc. 
259 East Erie Street, Chicago 11, 111. • 

Item: 2 Parts, musical memory, rhythm. Musical Memory: ' 
Subject listens to a number of pairs of 2 bir melodies played, 
on the piano and compares them as to sameness, or'change 
1n notation, key, or tempo. 
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Music 



McAleavey, Grace Anne 

GROUP TEST OF MUSICAL ACHIEVEMENT FOR THE THIRD GRADE 

'"'Pitch, rhythm, age, aural identification, vfsual note reading, 
. standard objective ... 
"Formulation of a Group Test of Musical Achievement for 
the Third Grade" by Grace A. McAleavey, Masters Thesis v 
Northwestern Univ., Evanston, 111. » • j 

» « 

Item: Group test of 4 parts. Tester reads questions aloud and 
S's raise hand to answer.. I. Knowledge of musical terms - 28 
mutiple choice items, II. Knowledge of syllable names - 20 fill- 
in blank items, III. Tonal recognition - 6 forced choice items,. 
IV. Rhythmic recognition 0 16 fill-in blank items. 

* J " 

. Scoring: Tentative nqVms established by percentile rankings 
of 300 children.- 



Validity:. Determii 
40 courses of musi< 



judgements of experts, and analyzing 



Reliability: De*termirted by correlating two forms of test 
yie'^bt correlation of \ 76 for 300 pupils.. 

Drake, ralelgh M. 

INTERVAL DISCRIMINATION TEST 

Timbre, age, aural discrimination, pair comparisons 



Drake, R.M., "Four New Tests of 
Psych. XVII, T933; pp. 136-147 

m < 

Item: Jest requires comparison 
Rlayed on the piano to determine 
longer or shorter than the first 
be compared. 



Musical Talent" , J. Applied 



of two musical intervals 
if the last interval is 
80 pairs of intervals to 



Validity: Validity coefficient fneasured by teachers 1 estimates 
.77 with N=46. I 

Reliability: Self-reliability coefficient - .83 with n*46. 



Taylor, Corwin H. 

MEASURES OF MUSICAL 



BACKGROUND FORM A 



ERiC 



Rhythm, melody, harmony, age, aur^al identification, background 
knowledge, understanding, successive categories, production 
"The Construction and Validation of certain Experimental 
Measures of Musical ; Potentiality" by Corwin H. Taylor. 
Doctoral Thesis Univ. of Cincinnati, 1941 

'Item: Consists of 4 tests: Test! I. Recognition of Tonality - ' 
a) * 32 melodies are presented and {must judge whether ..each is 
minor.or major key. b) 52 chords are presented and S must 
choose major cfr minor. Jes% II. Rhythmic Discrimination - 
a) 32 selections are pre'sehted arid S must indicate meter of each- 
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Music 



Realiability: Kuder-Richardson formula- 20 yielded a coefficient 

of .78 based on the test results of 2314 subjects from various . , 

size schools in Colorado. 

* 

Hooper, Sharon' 

- MUSIC READINESS TEST . ^ * • ' '> 

' . • - * Y A ~ ■ . * , '* 

Questionnaire, age, background knowledge, understanding, visual , 
note reading, standard objective 

Item: Test consists of ll/ items, divided into five sub-tests (. 
dealing with interpretation of terminology used in notation., \ 
ability to determine titles of familiar songs from a few measures 
\ * ' of .notation, concejjts'and skiHs concerning rhythm/ 

p Scoring-.' Norms given in means and standard deviations based 

dn test results of 200 students in grades 3 and 4. 

: Rel iability: Coefficients derived frdm test results -of 200 

subjects in grades 3 and 4. Split-half .= ;9T, Kuder-R,ichardson 

Formula 20 = .937. ". f ■ . 

Seashore, CarKEl. • Lewis, D. ' Saev'ei-t, J.G-. . 
SEASHORE MEASURES OF MUSICAL -TALENTS, REVISED EDITION 

Pitch, rhythm, volume, melody, musical -training, age, aural 

discrimination, pair comparisons ' '•*».. 

Psychological Corporation, 304 East 45th Street, New York V7,^N.Y. 

Item: Test consists of six sub-tests requiring the subjects 

. discrimination of pitch, intensity, Wime, consonance, tonal 

memory, and rhythm. . ..'..* 

"T" ' ' * . 

•Scoring; Scoring key furnished. Norms given- in percentile. 

/ * • • . *• » - . 

Validity^ The. authors present considerable information regarding*--, • \ 

validity and reliability injthe manual accompanying the test. . \ • 

Watkins, John 6. Farnum, Stephen E. * . 

WATKINS-FARNIH PERFORMANCE SCALE: ^STANDARDIZED ACHIEVEMENT . 
TEST FOR -ALL BAND INSTRUMENTS 

.M'usic score, musicaj training, brass playing, string playing, / 
woodwind playing, percussion p.laying, instrument playing w 
Hal Leonard Music, Inc. 64 E^2nd StV. Winona, .Mich. 

'Item; Consists of a series of musical exercises of increasing 
' difficulty presented to the student for Sight-reading- and 

• \ .individual. playing/ ' . * . . • 

Scoring:'. Any and each error in. pitch, tempo, rhythm, etc. 
incurred while the subject plays each piece. counts as minus 
1 towards, the total score, Norms given based on the number . # . 
of years the instrument has been studied.- * 
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^ x • r '/ A BIBLIOGRAPHY 'SROM A. N. GPPShiEIM'S 

. ' QUESTIONNAIRE DESIGN AMD' ATTZTCBZ MSASCa3MENT' 

X . Pro.bleme of Survey Design i y * w . V 

Sampling Methoda 

• 4 

1. Ae'Stuatft* Basic Xdftii o£ Scientific Sampling (London: Griffin, 
V 1962). . 



An excellent, nonmathematical account. 

i - 

* • • * * 

2. leldor Chain, "An Introduction to Stapling, " Appendix 3 in 
\ Claire Selltlz, Marl* 4ahoda, Morton Deutsch, and Stuart U. 

«. Cook, eda. , Jteaearch Methods in. Social Relatione (New York: 

> Holt, 1959). K 

3. F. Yatea, Sampling Mathoda for ' Consuaaa and Survaya (London: 
Griffin, 1949). 

- * • 

4. Morrla^H^Hansan, William N. Hurvitz, and William C. Mordon, . 
Sample Survey Mathoda and Theory (New York: Wiley, 1953), 
Vol. l % 

Practical sampling mathoda applied to survaya • 

^ Experimental Design • j 

5* Ronald A* Fisher, The Design of Experiments (Edinburgh : Oliver 
; and Boyd,;i£47). ^ < * 

A classical textbook on experimental design. 

6a We Ge Cochran and G*M. Cox, Experimental Designs (Hew York: \ 
• * Wiley, 1950). w . ^ 

-\ /7e A.* Ee Maxwell « Experimental Design in Psychology and the Medical 
^ a Sciences (London: Mcthuet^, 1958)^ . 

8. Allen T.. Edwards. "Experiments: Their Planning and Execution." . 
^ in Gardner Lindzey, ed., Handbook of Social Psychology 

V {Cambridge, Mass.: Addiaon-Wcele*, 1954J. 

9.. D. R. Cox, Planning o'f Experiments (Newark: Wiley, 1961) 1 . 

i, 

. k 

. « . ^ Purvey Design ^ 

< - "* 

• * 10. Claue A. Moser, Survey- Methods in Social Investigaticn (Londbn: 
Helnecacn, 1958). 

* ' A good genera; text on survey aatgpda. See cepeclelly 

, * . .^charatara on sampling awfc on interviewing . 
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11. Herbert\H. Hymen , Survey Design ^1 Xpalvala: PrlnqiplejJ £*S23L> 
/" antt Procedures (Glencoe, Ill~rT / "?ree Press, 1955) .^F 

For the advanced student. 

12. Robert L. Xahn and Charles F.'fcin&eil, The Dvjfraarfgs'of Interviewing 

(Haw York: Wiley, 1957). 

A uaaful textbook on surveys, questionnaire -design, 
and interviewing .' 

i 

Matching Problttt^ 

13. William A. Belson, "Matching and Prediction on the Principle of 

t Biological Classification," Aaolled Statistics . Vin (1959), 

> 65-75.* _ 

M » ♦ , * , 

On the uses of the stable -correlates method of matching. 

14. W. 2. Billewicx, "Matched Samples in Medical Investigations," 

British Journal of Preventive and Social Medicine, XVIII 
(1964), 167-173. ' 

It 

I nterviewer Errors 

15. Herbert H. Byman £t al,. , Interviewing in. Social Research (Chicago; 

University of Chicago Press, 1954). 

Experimental studies of interviewer bias and the effects 
of interviewers on respondents.- 

16. Robert L. Xahn and Charles P. Cannell, The Dynamics of Interview- 

Is*. Otew York: Wiley, 1957).. * 

See eapecUlly the section dealing with interviewer biee, 

Bgj^dte togLJBtf Sfifilll Medicine 

17. J f Morrie, Utet of Epidemiology (Edinburgh: Livingetona, 1957)\ 

A short basic textbook. 

tail Studiae 

18. Morris Rosenberg, Wagner Thleleus, and Paul tasarsfeld, "The Panel 

Study,"' in Marie Jahoda, Morton Deutsch, and Stuart W. Cook, 
dds,. RSaeareh Methods in Social Relations (Hew York: 
Dryden Press, 1951), Part It. • 
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19. James WVB. Douglas and J. M; Blomfleld, "The Reliability of 

Longitudinal Surveys*" The Millbank Memorial Fund Quarterly . 
XXXIV (1956), 227-252. T 3 — 

FrobltM-of Questionnaire Design k 
Interviewing 

s i. Robert L. Kahn and Charles F . Cannell,,The > Dynamics of Interviewing 
(Hew York: Wiley, 1957). ^ 

Chapter 6 is a useful introduction- to questionnaire design; 
chapters 7, 8 ind 9 deal fully with interviewing skills, 
sources of bias, and so on. 



2. Herbert H. Hymen et al. , Interviewing in Social Research (Chicago: 

University of Chicago Press, 1954). 

* 

. Experimental studies of interviewer bias and the effects 

of interviewers on respondents. 

3. Robert K. Merton, Marjorie Fiske, and Patricia L. Kendall, The 

Focused ^Interview. (Glencoe. 111.: The Free Press, 1956). 
/ . i 

A careful and clear analysis with many examples of inter- 
viewing problems and procedures. / 

4. Paul B. Sheatsley, "The Art of Interviewing and a Guide to Inter- 

viewer^ Select ion and Training," in Marie Jahoda, Morton 
Deutsch, and Stuart Cook, eds., Research Methods in Social 
Relations (New York: Dryden Press, 1951) i Part H. 

« 

An introduction to the conduct of interviews and the 
problems of interviewer training. 

5. Sleauor E, Mac3&t*^and Nathan Maccoby, 'The Interview: A Tool of 

Social Science, 11 in Gardner Lindzey, ed«, Handbook of Social 
Psychology (Cambridge, Maas.: Addisoto-Weeley, 1954). 

Basic textbook reference on problems of research in 
interviewing . . 
Mail Questionnaires 

6. Christppher Scott, Research on Mail Surveys," Journal of Roval 
' Statistical Society . XXIV, Series A (1961), 143-193, 

An excellent contribution to research IntoWll question* 
> , ^ naires, together with -a full review of the literature. 

* Question Saounc*^ ^ 

7. George Gallup, Qualitative Measurement of Public Opinion : Thf 

Quintamenslonal Plan of Question Desig n (Princeton, H. J # : 
.Aaerlcan Institute of Public Opinion, 1947). " 
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7. William A* Bel sen, Studies in Readership (London: Business Publi- 

cations Ltd., 1962). ' ^ 

A carefully conducted follow-up inquiry into the causey of 
unreliability in a national readership survey. 

Occupational Grading ^ 

8. Hilda I. Himnelveit, A. N. Oppenheia, and P. Vines, Television and 
the Child (London: Oxford University Press, 1958). 



j 



See pp. 82-84 for problems of obtaining father 9 a Job 
information from school children. 



9. Fortune Surveys, Jhe People o£ thft United States — a Self-Portrait 
(Fortune > February, 1950, Vol. 21) • 

10. Richard^ Centers, The Psychology gJL Social Classes (Princeton: 

Princeton University Press, 1949). 

^ "'' ' 

Eabarrasflng Queatioha „ 

^ ■ 

11. Alfred C. Kintey, W. B. Poaeroy, and C. B. Martin, Sexual Bahavlor 

in the Buean Mala (Philadelphia and London: W. B. Saundare, 

1948). , - , 

Saa aapaelally pp. 35-82 on interviewing technique,*. 

12. Allan J. Barton, "Asking tfaa Embarrassing Question," Public Opinion 

Quarterly, mi (1958), 67-68. 

A humorous set of suggestions for wording embarrassing 
questions. 

Reliability and Validity 

13. Lee J. Cronbach, Essentials of Psychological Testing (New York: 

Harper, 1960). - * ' 

,% A useful textbook on testa and their uses. See eapec^lly 

chapter 5 on validity, and pp. 126*142 on reliability. 

..14. T. W. Adorno, Blse Frental-Brunswik, EL. J. Lavinaon, and R. M. 

Sanford. Iha Authoritarian Personal*^ fltev York: Bar par, 
1950). ^ 

Checklists, Rating Scales, and Inventories 
Psrchological Measurement 

^ln^Barold Oullikaan, Theory of, Mental Taata (New York: • Wilay, 1950). 

A baaic taxtbook on test -construction theory. .J WSt 



( 



• ■ J 

13. Ann* Anastasi, Psychological Tasting (New York: Macaillan, 1963). 

Attitude Stateaents , ' » 

Attitude Theory 

1. David Krech, Richard S. Crutchfleld, and Eg«rton L. Ballachey, 

Individual In Society (New York: McGraw-Hill, 1962). 

See Chapter 5, "The Nature and Measurement of Attitudes, " 
for the characteristics of attitudes and attitude systems, 
t > ah exposition of scaling methods and types of survey 
questions. 

2. D. Katz and E. Stotland,' "A Preliminary Statement to a Theory of 

Attitude Structure and Change," in S. Koch, ed., Psychology ;* 
A Study of a Science . Vol. 3: Formulations of the Person and 
the Social Context (New York: McGraw-Hill, 1959). 

A videiranging essay on attitude theory. 

3. ; Hsns J. Eyaenck, The Psychology of Politics (London: Routledge 6 

Regan Paul, 1954). 

See especially Chapter 4 on the organisation of social 
attitudes. 

4. Edward A. Suchman, "The Intensity Component in Attitude and Opinion 
Research, 11 in Samuel A. Stouffer, ed., Measurement and Predic 
tion (Princeton, N. J # : Princeton University Press, 1950). 

5. T. Adorao, Else Rtokel-Brunswik, D. J. Levinson, and R\ N. 
Sanford, Jhe Authoritarian Personality (New York: Harper, 
1950). f ^ 

A classic study of prejudice, shoving the relationships 
between different kinds of prejudice and the development 
of the concepts of ethnocentrism and Authoritarianism. 

6. E. Wellin, 'Vater Boiling in k Peruvian Town," in B. D. Paul, ed., 
Health. Culture and Community (New 'York: Russell Sage 
Foundation, 1955). 

7. Bernard M. Kramer, 'Dimensions of Prejudice," Journal of Psychology , 
nra (1949), 389-451.* " * jJ/T 

Writing Attitude Statements 

8. * Allen L. Edwards, Techniques' of Attitude Seals Construction (New — r 
York! .Appleton-Century-Crofts, 1957) . • 

See in particular p. 13 showing how to edit attitude 
statements. 



.4. S. S.. Stevens, "Mathe«*tics, heasuveaent, and Fsychophyslcs 

In S. S. St*v*ns, ed.. Handbook ol Exnarl-ntal Psychology 
(Hew York: Wiley, 1951). " 5smfc 

On general aeasureaent theory. 

5. A. I». Idwards, UfihBifluog of Attltuda-Sc«l« Construction 

(Mew York: Appleton -Century-Crofts, 1957). 

6. Madid* W. Riley, J. W, Riley Jr., and J. Toby, Sociological 

|£a4itt aft SfiaU Analvslf (Haw Brunswick, M.J.: Rutgers 
University Prase, 1954). - 

7. W. s. Torgerson, Thaory and Methods of Scaling (Haw York: Wiley; 

a 1958) • 

For the advanced student. 

8. Leonard W. Ferguson, Personality Meaaureaant (Mew York: McGraw- 

Hill, 1952). : ™T , 

. Sse chapter 4 on attitude aeesureaent. > 

9* J. P. Guilford, £sv£hos£trlc Mjtbgdj, (New York: McGraw-Hill, . 
W54). . 



See especially chapter 16 on factor -analysis, chapter 
4-7 on psychophysical aathods, and chapter 15 on Iteav 
analysis.' 

.'.»'•(' 
10. Lee J. Cronbach. Essentials of Psychological Testing fMav York: 
Harper, 1960). " 

A deeful textbook on tests and their uses. See especially 
chapter. 5 on vaHdlty and chapter 9 on f eotor-analysis . 
Percentiles are discussed on pp. 75-78; standard scores 
^ - on pp. 78.-87; calculating correlations on pp. 110-125; 

\ problesjs of reliability on pp. 126-142. 

Soclal-Diatanca ggajej 



11. I. S. Bogardus, "Measuring Social Distance. ' n Journal of Applied 

Sociology. IX (1925), 299-308. . 

12. I. 8. Bogardus, "A Social -Dlatance Scale," Sociological and Social 

gejfj£ch, XVII (1933), 265-271. "7 " 

13. Harry C. Triandis and Leigh M. Trlandis, "Race, Social Class, " 

Religion, and Nationality as Determinants of Social Distance, 
JajgMi SL Afeporael *e4 Social Psychology. LSI (1960), 
110-118. v. • \ 

. ' rj^~7. ' 

Use of a revised^ version of social-distance scale, 
developed with the aid of the Thurston* successive* 
< Interval procedure and checked for linearity, uoidlaan- 
slonallty, reliability, and reproducibility. 
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Factorial Seal., l-^ - 

22. Raymond B . Cattail, Factor Analy^s 'fNett Yorkl Harper, 1952)V - 
***** • * » 

23. BjiM .J* Eysenck, The Psychoid oj, Politics ^London; Routledge 
% - / Kegato Paul, % 1954).«i - ^\ - 

; • See especially chapter 3 on scaling procedures an<k valid* 
tdop and chapters 4 and 5 on factor -analysis applied to 
u attitudes and their clustering, 

24. Hans J, Eysenck, "Primary Social Attitudes : n A Com>arison of 

Attitude Patterns in. England 7 Germany", and Sweden," 
Journal of Abnormal and Social Psychology . XLVUI (1953) t 
563-568. ✓ 

A cross-national comparison of attitude structures with 
the aid of factor -analysis. V m * 

25. Earl S. Schaefer, "Converging Conceptual Models- for Maternal 

Behavior and for Child Behavior," in John C. Glidewll, ed. , 
Parental Attitudes and Child Behavior" (Springfield, 111.: 
Charles C. Thomas, 1961). • . 



A presentation of the two main^omponenta of maternal 
attitudes, in terms o^ factor -analysis. 

s 

»26. Louis Guttman,' "The Basis for Scalogram Analysis," in Samuel A. 

Stouffer, ed., Measurement and Prediction (Princeton, K.J. : 
Princeton University Press, 1950). 

J • ' 

*H The main source book for the Glttman scaling technique 
and its theoretical background. 

, i 27. L. Podell and J. C. Perkins, "A Guttman Scale for Sexual Experience 
"* LIV a957)! 1 42o"22!° te,n ~ ^ SSS5Si 222£holoat, 

28. Paul Wallln, "A Guttman Scale for Measuring Women's Neighbor lines., 

The American Journal of Sociology . LIX (1953), 243-246. 

29. Joseph Trenaman, "Cuttman's Scalogram Analysis of Attitudes,' 1 in 

Attitude Scaling (London: The Market Research Society, 1960) 

A useful and critical exposition of the scelogram-board , 
technique. " ( * 
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* 39. Kurt Levin, Principles g£ Topologica l fevchofogy (New York: McGraw- 
. . H1U, 1936). 

Levin' • definition of behavior ea B - f (P,K) end its 
% '- inplicatlone. 

/ 40.\ George G. Stern, "B » P (P,E), fl Journal of Pro jective Techniques 
end Personality Assessment. (1964), 161-168. 

o > 

• , * An attempt at studying environnental influences on 

^ * perception, attitudes, and behavior. 

\ "41. Gardner Murphy, Lois B. Murphy, and Theodore M. Newcoab, 
f Experimental Social Psychology (New York: Harper, 1937). 

/ See especially thei r co mments on the relationship between 

> attitudes and behavior on pp. 898-900. 

J \'/. ° '* 

Projective Techniques in Attitude Study 

(yatieral feeference»^to Projective Techniques „ * 

1> ~ Donald T. Caapbell, ''The Indirect Assessment* of Social Attitudes,! 9 
V/ Psychological Bulletin. XLVH (1950), 15-38. 

A very useful review of projective methods in attitude 
- * -V . . research; particularly thorough, on "information 11 tests, 

guessing tasks, tests of critical thinking, and pictorial 
*- ' devices • 

2. »R. H. Anderson and Gladys L. Anderson, eds., £a Introduction 
Projective Techniques -flfaw York: Prentice*Hall, 1951). 

• • X basic textbook; * 

^ J * Sell. B Pro lecti^ra Techniques (New York: Longmans, Green, 

(Steer al textbook. * 

fe4^;tellti£^ Deutsch and Stuart W« Cook, 

' Research Methods in Social Relations (New York: Holt, 1959). 




Sea chapter -8 on*proje<ttRfcve and other indirect methods* 

. • — i . + 

5. He Henry , Motivation RaaearcK (London: Crosby Lockwood & S?n> 1958). 

Chapter, ^vdeal-s with; the use o£ projective techniques in. . 
surveyed * 

6. 6. H. Smith, Hpfcivation Waeercfr-in Advertieing end Marketing 

(Nev York: KcCrav-HillJ 1954). 

>^ • f * '- « ''♦v.*' 5 k ' — 

Includei~a review of- projective tec£nlqu#s uaed in eerket x - s 



15. John W. Atkinson, ed., Motives in Fantasy . Action, and. Soeiatv 

(Naw York: Van -Nostrand , 1958). 

A series of Studies using* TAT measures of th. need* for 
' achievement, affliation, and power, complete with scoring 

system..^ . 

16. W. Caudill, The Psychiatric Hospital as a Small Society (Cambridge 

Mass.: Harvard University Press, 1958). 

See especially the use of drawings of hospital scenes as 
projective devices (Figures 7-3 to 7-6). 

*17. M. Haire, "Projective Techniques in Marketing Research Journal* 
" . • ■ of Marketing . XIV (1950), 649-656. « ' 

Shows the use of shopping lists as a projective device, 
. vto obtain the "image" of the housewife who buys instant 
coffee. *V * 

18. Re Proshansky, "A Projective Met&od-fot^e' St,udy of Attitudes • 11 

Journal of Abnormal and Social Psychology . XXXVIII (1943), 
383-395. ; * 

A pictorial method for studying attitude to labor. 

19. M. Shepherd, A. A. Cooper, A. C. Brown, and G. We Kalton, 

Psychiatric . Illness in General Practice (Hew York: Oxford 
University Press, 1966)* * 



Pseudofkctual Questions 



20. B. L. Hartley, Problems in Prejudice (New York: Kings Crown 

ft Press, 1946). ) « ' \ m # ; 

• Seja>>«8pecially the use of* "none-such" groups in attitude 

research, e.g.,/Wal&ni*he% Danireans, and Pireneans. 

21. Theodore M. Newcomb, Personality and Social dnahge (New York: 

" Dry'den Press , 1.943) . " '. 

An~ early use of the ''Guess-Who" technique. 

• v ' " . 

Play Techniques.. t • : * • ~ * * „ 
» V7 r' '• - • .* • • ► 

22. . Hugn^artshorne and Mario A. Mavj Studies in Deceit (New York: 

Macmlllan, 1928). . ~ ' 

• *• : ' , ft * 

Experimental. studies of children's attitudes. 
./ * * . * * ' • r • ? ' 

- 23. L. E. Ucko and Terence' Moore, "Parental Roles' as Seen ly Ypung , 
. ... k . ; Children .in_Doll Play," VltA Humana , yi (1963) , 213-242. 



4. Gardne^Lindiey and Edgar F. Borgatta, "Sociometrlc Measurement , M 
in Gardner Lipdasey, ed., Handbook of- Socia l Psychology 
(Cambridge, Mass: * Addtson-Wesley , 1954) . 
* . * * - * * *^ 

An excellent textbook reference on the^whole field of 



socionetry , 



5." Robert J. Wherry §nd Douglas H** Fryer, "Buddy Ratings; Popularity 
Contest or leadership Criteria? 11 in J. L. Moreno, ed., 
The Socionetry Reader (Glencoe, 111: The Free Press, 1960) . 



Semantic Differential 



Tannenbaum, 

L , _.. . University of 

Illinois Press,H957>: . *' * 



r/ **\ ^£ ^Charles $. t(£s$pbd, George J. Stfci,'*aqd Percy H. 1 
/ ( Thq Measurement of Meaning (Urbanadf ill, : 



Introduces the technique of the semantic «Kf erential • 
See especially chapter # 3 ,M The Semantic Differential as a 
Measuring Instrument,'' and. chapter 6 "Semantic Measure* 
_ ment in Personality and Psychother apy Res earch/ 1 

/ 7. J; J. Jenkins, W. A* Russell, and J. Suci, "An Atlas of Semantic 
Profiles for 360 Wqfds," American Journal of Psychology . 
LXXI (1958), 688-699. , 

Repertory Grid 

8. ' George A. Kelly, the Psychology of Personal Constructs (New York: 
' - V. W. Norton, 1955). 

ft Volume I cdrirtfaine the basic theory, and the development 
of the repertory test (see especially chapters 2*, 3, and- 

; \ 5).* fc ; ^ r \ « 

9* D* Bannister, "Personal Construct Theory: A Summary* and Experi- 
mental Paradigm," Acta Psychologies . XX (1962), 104-120. 



r A clear, expository paper. 

10. D. Bannister , IA Ihe Nature and Measurement of Schizophrenic thought 

Disorder , 11 Journal of Mental Science . C VIII (1962) , 
825-844. 

11. L. M«« Levy And R. D. Dugan, "A Factorial Study of Personal Con- 

structs, 11 Journal of Consulting Psychology . XX (i956>,^ 
* 53-57. , h : 



Stereotype* 



12. V* Buchanan and H. Cantril, How Nations See Each Other, (Urbane, 
( , Illinois: University of Illinois Press, 1953). 

Studies of the role of atereotypee in International 
attitudes. 



19 - 



J. P. Guilford-, Psvchomattc M»fh..» , . McGyav-HlU, 1954) i 





''^H^?^ S Atlattcal „ „ '^cj^nal ga* e arch (Nov 
* • Tprk: ''Houghton Mifflin, ' •» ~ ** — 



C#U * t ;J5 C0l i^8 h «d Loche Va« At&, itlsOc^ tonceots (Nov 
«ork: McGraw-Hill, 1963). . ~ H " , p v 

A uooful programmed learning feeac. 
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Why an evaluation plan? 

I A plan for evaluation may be viewed as a blueprint 
rSSE prov,des Intormttlon about the intended 
jacttntfes (and implicit, philosophy) of an evaluation 
pwtayor. A plan allows the evaioator to explain his 
Conception of the evaluation tasn to the client be- 

funlty"!! SmST^ 8 **** ,h * n has an opoor " 
intended actions _._ 
feels this is. necessary. 
, Certain elements should be taken into, account in 
designing any evaluation plan, but eacn plan must 
necessarily reflect SVuniqueness of the particular 
program being evaluated. The commentary and 
checklist featured- in the following paces suaoest 
aome design considerations. . * 99 



unity aq see if he approves oi the evaluator's 
s and may restate his concerns-* he 



K**an Brophy is I graduate assistant in the College of &Suc»- 
tton^at the Unrverwy-of Illinois at Urtw^Oiaraoaign. Arcsn 
'» asaodate oean for researcn eno rfrftee in the 
oonege of Education. Oennis Goowr * an alumnus of the Cof 

SE^SS" *H * ^f" 1 * e*"**™ o' the ATM Of 40. 
aructlonai Technology at Syracuse University. 



( ' " I * Why evaluate? 

in general, evaluation is conducted in response to a need 
expressed toy people within a program, or individuals 
• or agencies interested in a program but external to 
it Evaluation 'may focus on what has been done, 
whft is being, done, and/or what might be done. 

Evaluation can assist In: 

Planning procedures, programs, and/or products; 

Improving existing procedures, programs, "and/or 
products; - 

Justifying (or not jusflfying) existing or planned pro- 
cedures, programs, and/or products; 

Evaluation may yltld: i* « 

Description* of procedures, programs, and/or pro- 

Judgments of the value of procedures, programs, 
and/or products. , 



Evaluating for program planning t 

Wtoat program preferences and needs*are held by 
various people and institutions^ , ; * 

Whet discrepancies exist between desired and actuaf 
status of various program elements? 

» **- 
What means are feasible for attaining desired goals? 

What will happen if .proposed goals are,attained?„ 

/ Who are thaadvocatesand adversaries oflhe proposed 
program? , , 

Evaluating for program improvement 1 

Are strategies working t as planned? 

What unanticipated results are aff&ctipg program 
operation? . 

Are program , issues being dealt with satisfactorily? 

s » t 

Is there sufficient program flexibility to meet new 
preferences or concerns? * • % 

Is th$ program content appropriate and effective? 

Are Internal/external relations hindering or enhancing 
program strategies? ' « 

Uttie program as effective as similar programs? 



Evaluating for program justification 

Are the goals of the program justifiable as viewed by var- 
ious people? 1 

What wis not attempted because of the existence of this 
* program? 

f*What exactly was done arid what were, the results? 

' Shooid the program be altered^ expanded, or disbanded? 

* * » 

What do advocates and^adver^ries want to Know about 
the program? 



Who is 



evaluation for? 



The people who m typically most interested in |p 
•valuation are'theyndtvKtuals or groups commis- 
sioning the evaluation, the persons involved in 
conducting' the program being evaluated, (he 

^program participants, and the supporters or de- 
tractors of the program. Special attention should 
be given to* ail these audiences, especially the audi* 
ertte that actually commissions the evaluation. 

' The evaJuator must be responsive to^eacn audienf* 
and recognize that different kinds of information 
may be desired by or useful to different groups.' ! ; 



Potential audiences or consumers 

ticipantr 

1 Teachers/re^uro* persons/faciiitat^ns 

Program administrators, implemented and sponsors 

Policy makers and public offtoals 

Community interest/action groups 

Academic dtsopiine specialists 

Technical and methodological specialists 

General public J . 

Other ! : 

(Specify)* 
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* What focus should Jhe 
evaluation have? 

SSSlKttSS^ bMbto to define tb«e 
^EST" im f»rt«nt influence 
o^tht^jidflc situation being evaluated. Tht delinea. 

■ of the client; the ; orientation of tht eyaiuator, end 
used Uy both to obtain Ma initial inform* 
^ ^5*?*? tiv * im POrtano» of variabiee may 
nZZ ZH* 09 "!! an ? /or "••"•fen proceeds. 

^■>^ dt *!I cw (mlu * tof - cti#m ' and others) ~- ' 
w<Qht D».aflactad b^the proeaaaaa and raauits i 

S^a!!** 1 " 111 " 6 * there wHi be many 
Pwan^imirwingvariabiaa than can bereeaa 

•«22CTS-£225^S2?r pto 'W^^tt.jaartaa^n^rtof 
priorrtj » will, differ ecroae 
W«^a<^oativ. affort» should' seek to iliu- 
S^ SLSfiL!^' •eonofrtfc, and/or political 
oortaxt within which tht program operate*, 
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Information desired by Interested audiences 
Program dynamics and procassss 
Program outcomes ' 
Progiam context and setting 
Program issue* 

• 

Information needed to make decisions 
Content 

•natructJonai/administrattv* strategies 
Student participant capabilities 
Personnel and institutional nsouress 

General categories of information 
Program, intents MS 
Oehaviora 
Interaction* * 
Inteueietionships 

CaueasMecta (lauding side or unintended effects) 
Coats' ° 

Successes^ 
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What should you know 
about the ©valuator? 

Any proceee used to determine the mdfit or worth of a 
program necessarily involves avaluator judgments. 
Judgmanta art ref lected in tha selection of variablee re- 
quiring attention, tha scurcee from which data are ot> 
'tainod, the techniques uaad to gather information, and 
tha meessgee finally convayad to an audience. Bacauaa 
so many judgmanta may ba made, tha dlarrt may wtahtp 
inquira into tha philoeophical and methodological orien- 
tation pf tha avaluator, his motivation for wafting towv 
duct an evsluatioa hie knowledge d the problem under 
study, hia toarianca. Wa capadty to work with people, 
and hia ability to report information. Such information 
about tha eviuator (similar quaationa might ba raised 
about tha dierit) pqpvkJee insight into (few a particular, 
avaluator may maKa judgmanta. Tha diantia than battslr 
praparad to select tha appropriate avaluator for tha 
specific program, procedure, or product to ba 



Practical background 
Experience in evaluation 
Professional axparianca 
GeneraJWork axparianca 



. Educational qualifications 
Academic background 
Exparianca in education ^ 
i. Professional intarsats 

OriantatJon 

Perspective on purposa of avaluation 
Servicefrictftloner 
JJeeeercMheoretidan 
Outcomes /pro cesses 
Prespectfiedfresponaive 

Mathodology / 

Academic discipline related (e-g., historical) 

Informal techniques 

Behavior analysts 

Judgmental 

Oeecriptiva \ / 

IntaraAn analysis 

4 

• Past standards . / 
Knowledge of other evaluations j [ 

0 Reputation y 
Detivery on what was asked, when needed 
trustworthiness 



Style 

Responsiveness to dient 
Personable 

Ability to work wfth Qthers 
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What are the moral, ethical, 
and legal responsibilities 
jot the evaluatcr? 

An evaluation may examine many criticaJ and ser&tive 
aspects of a program* Consequently* a number of 
conflicts 'can arise among clients, equators, and au- ♦ 
dlencee.*An evaluation plan should reflect an aware- 
ness of potential problems, and should specify to 
which existing information (e.g., student records) the 
evaluator may have access, and wnat he may do with 
such information. Agreement should, be readied as to 
anonymity of sources of information — when such 
anonymity is 'deemed appropriate* The plan should 
stipulate m deeny as possible contingencies on deliver- 
ing the promised information, by a' predetermined 
date. There should be agr*ment as to who will have 
acceee to the evaluation report and who will not All of 
• these elements affect the way in which an evaluation * 
may be conducted and reported/ and the obligations 
incurred by both evaJuator and client 



Establish guidelines 

Penalties for not meeting expectations (especially 
deed dnee) ^ i 

Extent of report reporting only what is asked for. or 
giving information beyond ongihaJ request 

Reporting detrimental information 

Acceee to evaluation information — who will, have ao» 



Authorityyof the ^valuator or client to determine 
what to report 

GeneraHzabiJity of results and implications 

Specify the client's parameters of power to ter- 
minate the evaluation process 

Conduct of the evaluation 

Use confidential information appropriately 

Avoid invasion of privacy ^ 

Obtain permission foil use of certain Inforniattoh 

i^Maintain open exchange of ideas jbetween tfient and 
evtluatorcorKeming each other's interests 

Tell participants in the program what this evaluation will 
iffcoJve an^rovide 
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How do you conduct J 
an evaluation? 



In an attempt to focus the evaluation, tha dient and 
©valuator typically define critical audiences for the 
evaluation, and the issues or variables to be ex- 
amined. A next step iff evaluation is an identification ' 
of the information relevant to each issue or variable, 
and the Hkely sources of that information (e.g., 
learners, citizen committees, class activities). Next, 
the evaJuator must determine how much information 
should be collected, and from what proportion of ♦ ; 
the data sources (which sampling procedures to 
use). Once these estimates have been made, ap- 
propriate instrumehts ind procedures should be . 
selected or constructed : and data gathered,, finally, 
tha comparison of data (including, judgments) with 
appropriate standards is also essential / 

Questions jbout timing of information collection, ex-\ 
ptanations as to why certain intoftnatton is^ being col* ^ 
lected, and means of coding and storing. Information , 
must also be considered. Th* conduct dian evaluation 
win interact with or intervene in the'conduct of a prc^ 
gram. Therefore, it is important that careful consider* 
tfon be given to the process of phasing jvaiuation into 
program operations. 



Identifying primary audiences 
Program funders 

Program managers or administrators 

Program participants 

Program or product consumers 

Others who may exert an influence on the program 
(&g^ content specialists, legislators) 

Identifying. critical issues / 

Outcomes 

Processes 

Costs 

'Consequences 
Justifications 

Dgta sources I 

People 7 f 

Existing documer/ts $t records " " 



Related research in evaluation 

Techniques for collecting data 

Questionnaires and ppinionnaires 

Interviews y 

Observation schedules 

Active participation in program 

Historical inquiry * * 

Standardized tests . 

Rating scales. 

Behavior analyses 

Checklists ^ V 

Attitude scales 

Interaction analyses ' „ • 

Utilization of biographic data 
Anecdotal records 

Techniques for gathering standards 
Statements by program personnel 
Statements by content experts 
Reports/recommendations by boards, commissions, etc 
Statements by regulatory agenoes 
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How do you report 
the evaluation? 4 



.People have.different abilities and different experiences 
which influence the way they receive and use evalua- 
tion reports. Both ©valuator apd client should take 
these differences into account when interpreting and 
reporting evaluation Information. Reporting standard 
scores from an achievement test for example, may be 
meaningful information to some audiences. For other 
audiences, infdrmation may need to be presented in a 
different Ifoim Similarly, the inferences drawn from cer- 
tain information may be useful and defensible when 
presented to one* group, but may be inadequate (or 
even have a negative effect^ when presented to another 
group. It is important to understand what criteria var- 
ious people use to judge programs, what standards are 
relevant and meaningful, what indicators ^people ao 
cept as legitimate, and in what form and in what 
language all of these things are to be meaningfully dis- 
cussed. Finally, the evaluation report must be timeiy — 
•there is often too little information too late. 
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Types of reports 
Written 

cm ■■/ 

PropressNeports 

Final reports 

General 

Specific 

Technical"* 

Nontechnical* 

Descriptive only 

Evaluative and judgmental — 1 

Makes recommendations 

Does not make recommendations 

Modes of display 
Case studies 

Portrayals ^ 
Graphs and charts ' 
Test score summarization* 
Scenarios 1 
Questions/answers 
Product displays 
Multimedia presentations 

t 

Simulations 
Dialogues/testimonies 
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How do you encourage people 
to use evaluation? 

The process of evaluation, as well aisthe utilization of 
evaluation results, requires Adear understanding of 
We social, personal, and political dynamics surround- / 
.ing any educational activity. An evaluation plan should/ 
show how the results of an evaluation will w 
into these dynamics, if any satisfactory use is to -be 
made of the evaluation information. / - 

People will be more likely to use evaluation informa- 
tion if they pprceive the process and/or resulting infor- 
mation as relevant to their needs, if they see the 
potential benefits deriving from the evaluation, and if 
they find the evaluation results to be available when 
needed. People who have been involved in/the original 
planning and implementation of an evaluation are more- 
likely to use evaluation results than those who were not 



/ 



[ Ways to encourage use 

To develop commitment involve people from inside and 
• outside the program in the evaluation process 

Report results when desired by project staff 

Indicate alternative courses of action 

Indicate implications of the findings 

Make presentation clear and attractive. 

Provide assistance for additional analyses of results as 
needed 

Provide technical assistance for implementation *>f eval- 
uation suggestions / 
Provide time for the study and use of evaluation findings 
Results should be valid 

Reporting of results should take into consideration the 
background and orientation of audience 

Reporting must show sensitivity to political/social factions 
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What does it cost to do 
an evaluation? 

Conducting an evaluation necessanty r*(;utrgs expend*- 
tore of resources: money, time, and personnel. M eval- 
uation plan should induce estimates of cost in terms 
of dollar outlay, time expenditures of stiff and others 
(such as those from whom data are sought), particular 
expertise needed, etc Evaluation may be more critical 
to some kinds of programs than to others; thus making 
^^•f*Jto«tions of resources legitimate in some 
cms. However, some form of evaluation can fit into any" 
budget Reasonable cost can be determined by an esti- 
mate of the significance of the issUes and the likely 
impact of the evaluation. The value of conducting an 
•valuation may also be>nfluenced by the.a\fcjiability of 
information about similar-programs, procedures, and 
products, and the availability of existing information on 
the program to be evaluated. 




Money * 
Evaluator f ees 

Trmrt and accommodations 
S uppfie s and communication 1 
Dissemination arxj implementation of findings 

Cost* to program personnel 

Time of personnel in gathering information about various 
aap^ps of the program 

Time and effort in facilitating evaluator requests 

Explaining and defending evaluation to program con- 
stituents & 

*> 

Considering, rejecting, or modifying evaluation sugges- 
tions 

Potential costs of acquiring Information 
Invasion of privacy' , % 

Interpersonal relationships affected negacvety 
Psychological influences on speofic incivtcuais 

Other difficulties v 

Formation of .defensive attitudes in people involved m 
the program f 

Negative reactions of audiences not constfered 

< # 

Controversy generated because certain issues net con- 
sidered * « 
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IX. ^COMMON EVALUATION HAZARDS AND HOW TO AVOID TUBl 

* . . • -s : . r- . . 

Each of the following* subsections provides a brief de- 
scription of a commoi) evaluation hazard ' and what should bo 
done to avoid it* The hazards are presented in the apj*roxi-» 
mate drd&r of their judged frequency of occurrence and soyur- 
ity. Not all hazards, however, are relevant to all evalua- 
tion designs . An attempt has therefore been made to describe 
ad early as possitxle in each presentation the situations 
where each hazard is likely to* arise* 'Unfortunately, because 
many of the hazards are relevant to two or more designs, it 
was not feasible, to organize this chapter according to evalu- 
ation "design* . X 



hazard 1: Claiming Much, Providing Evidence of Little 

k 

The goals of an innovation may 'be multiple and gratjd;, 
t*he svidsnce may be modest and limited, as in one instance 
where preventing school failure through increased parerit in- 
volvement, home/schbol codperation, building positive atti- 
tudesj and academic gains were claimed 4 while- the only data 
were pre-post measures of spelling* 

j ■. 
In another ' gubr.iss'ion, if wasrdaimed that participants 

would get in less trouble with the law. Evidence for this 
claim was not included 'in the submission; in a' 'discuss ion; s 
the claim turned out to be based on the fact that "The $tiil w 
' dren are being watched from 8:30 to 3:30 every day. This 
W9uld help keep them olit of trouble. 11 In the same submis- 
sion, the claim that regular progress during the fall-spring 
year resulted from the Sumner program turned but to mean "If 
our children gained three months in six weeks, they'dj)e 
gaining the equivalent of one year in eighf months." 

The panel can only reach Judgments based on the evi- 
dence; whpre evidence matches the claims, a favorable deci- 
sion is far more likely than 'where evidence falls far short m 
.of "goals, objectives, and claims; 
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Hazard 2:> Selecting Measures Hot Logically Related to 
the Intervention 

Valuable a3 standardized achievenent tests nay be for 
assessing nany interyentions, in' nany instances there are * 
,othet>, vsozk .sensitive, nore appropriate sources of data. 
Measures should be selected because they will assess the cen- 
tral consequence of the intervention, be it inproved career 
decision making, adequate knowledge of human reproduction or 
contraception, inproved consuner skills, mininun conpetencies 
£&r self care in dressing and toiletting, increased pride .and 
self confidence as a nenber of one's ethnic or racial group, 
inproved teacher classroon control, or a better school plan- 
ning and budgeting systen. 

For alnost all educational outcones, reliable sensitive 
measurement is possible, using ^techniques as varied as analy- 
sis of basic, school records, interviews, questionnaires, case 
studies, observations, and tests. , 



Hazard 3: The Use -of Grade-Equivalent Scores 

Grade-equiyalent .scores' present *a severe nethodological 
problen ,when used in no ra-refe rented evaluations. No norn- . 
referenced evaluation based on grade-equivalent gains can 
be accepted. When used with control-group or regression- 
evaluation no.dels, the methodological pfcob lens' are far less 
severe although problems of interpolation renainl 

Grade-equivalent scores provide an insensitive, and, 
in sone< instances*, a systematically distotced, assessnent 
of cognitive growth. The concept of a grade-equivalent *.core 
is misleading — ^or exanple, a grade-equivalfervt score of five 
attained by a third grader on a nath test does not mean^that . 
he knows fif th-gr^de ,nath# Possibly he can do third-grade 
nath as well as the * average fifth grader, but it is likely 
that no fifth-grade students have ever taken the third r gradc 
level of the test. 

v * * s 

* t 

The use of grade equivalents for- evaluation purposes 
creates- a second problem in that they dofnpt forn an equal- 
intervrrfrscale and should n£ver be averaged. Finally, grade 
equivalents are constructed based on • the assumption that 
-growth occurs at the sane rate throughout >-the school year. 




5 



107 

\ , 




for coDputatlons and thH L? ^ . " P"" 11 """) »<! used 
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the norm gfoup. When control groups are available, few eval- 
uators would consider testing the treatnentrand control 
groups more than a few days fipart.' Uhen norrts are used as 
a substitute control group, this same consideration needs 
to be given to testj dates. 

• To; avoid this hazard, .treatment group students should 1 
be tested within two weeks of • the midpoint of the interval 
during/ which the normative data were collected. Particularly, 
jwhere the evaluation sp^^ns a summer, both pre- and posttest 
should deviate from the norming date in the same direction 
-otiu by the same amount of time.. Testing within six weeks * 
of empirical normative data points is permissible if linear # 
interpolations or extrapolations of the hormat^ive data are 
mad%. 'Tests that *provi£e fnormative data *f or only one point 
in the year should not be used in fall-to-spring norm-refer- 
enced evaluations. The principles of comparability in tim§ 
of data collection also apply ,to other outcomes, such as jok 
placement rates that' will show sVstematic seasonal variation. 

; fca£ar<i 6: The Use of Inappropriate Levels -of Tests 

If most of the pupils achieve very high or very low test 
scores, the level of the test may be inappropriate for" as7 
•sessing th^fr perf ormance. If pupils encounter the test floo 
at pretest tine or the ceiling at posttest tim^, treatment J 
effects will bg underestimated. Conversely, if .rhe o6iling ( 
is encountered on the pretest or the floor on the posttest, 
gains will'be overestimated. Ideally, student* should scorfe 
in the middle of the range of ppssible r^w scores; Average 
performance for groups of students shpuld generally fall be- 
tween 30 and 75 percent y£ the total number of items answered 
correctly. 

To avoid this hazard, test levels should be selected 
on the basis of the achievement levels of the students, not 
on the basis, of ;their grade in school. In. most cases, the 
nominally recommended test level or one level above or below 
will be suitable for testing "most groups of students.^ 

Using a test level other than that nominally r^fommended 
for a particular, grade is likely to mean that norms « tables 
for the tested students are not included in the test manual 
for that level. However, it is not meaningful tor assess 

♦ # ' 
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either status or gains by comparisons* with , students at a dif- 
ferent grad,e level. The status of a sixth grader J should be 
•assessed -using sixth-grade noras even if he is tested with 
a\fourth-grade test. Most^majox test 'publishers , fortunately', 
have interlocked, their test levels by providing an expanded 
standard score scale which enables ttte determination of score 
cmulvalehcies between adjacent test levels, ifiese scotes ' 
m*k~e- it possible to predict from a pupil^s score ont^one test 
level h<?w he would have scored, on the, next higher br lower 
level,, thus providing access to the in-level norma. Another 
possibility (although rare) is that the manu^'may provide- 
normative data for pupils tested out of level. " - 



.J 



Hazard 7; Missing Data 



Regardless of the evaluation* model used, data analysis 
should be based only on those. Students with both pre- 
"treatment and post- treatment scores Interpretation of these 
4a ta, however, should take into account the characteristics 
of the students who dropped out, entered late, ot graduated 
from the project. Tor example, if pll of the lowest scoring 
students oti a prerest^ropptfd out before posttest tine, the* 
average posttest' score would increase with respect 'to the 
pretest scores simply because of the missing stiyients. This * 
increase could be^sinterpreted as *a gain. Like^se, if • ' 
the .high-scoring* students graduated £rom the group, the mean 
posttest score would 'be artificially deflated*.. 

To avoid this hazft*d u every effort must be made tO/Ob- 
'tain'pre-tjreatment and post-treatment data for each project 
participant j and to base comparisons on those 'students- for 
whom both data elements are available. Data from students - 
having ot^ly pretest or ttily posttest shores must be carefully 
examined *to see if thfey differ in some systematic way from, 
the data of students having both pre- and posttest scores. 
A description of any of these differences should" be included 
dn tljie^eyaluation report. ' 

• i « 
* , ' t 

Hazard 8: The* Use of Noncomparable Treatment and *1^J:!^. 
fC Control Groups ■ ' ~* ~" ~ " ™" . p 

•* «. 

In ^trvfe 11 experimental designs, treatment and control 
groups should be similar in all educationally relevant 
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respects before the ^ypeatmerit begins. Groups that differ in/ 
•terras of pretest scores present an obvious source of bias. 
Other more subtle factors such- as differences in. age, son, * 
race; or socioeconomic status can also exert strong biasing 
influences and should be avoided. Nonvolunteers should never 
be used as controls for pupils who volunteered (or were vol- 
unteered by their parents) for a particular instructional 
treatment. 

Whenever possible, students should be assigned* to* treat- 
ment and control groups on a random basis. For exahple, with 
a , semester-long reading program, pupils could be Randomly 
assigned to first- or second-semester, groups. For the flr»t. 
hal.f of the year, one group would serve <as the control group 
for the other, but both.groups would ultimately receive equal 
amounts of the treatment. : 

■In some cases, pre-existing groups Will be enough alike 
so' that they can appropriately be considered equivalent to 
random samples from a single population. In other cases, 
a control group will be known to differ systematically from, 
the treatment group. Where the difference is, small, the 
control groijp model ^y still provide the beat metho^ of 
evaluating the project, and statistical adjustments can be 
made to compensate for between-group differences. '-Where the 
differences are large, 'however, there is no ; way in which a 
noncomparable control group can provide an accurate estimate 
of how wellfthe treatment group would have done without the 
treatment. 



Hazard 9: The Use of Inappropriate Statistical Adjustments 
with tJonequlvalent Control Groups * 

There are several statistical jffrocedures that are widely^ 
used itt an attempt tb compensate for* initial differenced be-* 
tween treatment and control groups., Some are legitimate 
while others are nq(. Making betweeh-group comparisons using • 
either "raw 11 gain / scores or "residual" gain scores fall; into*, 
the latter category.^ Both procedures should be scrupulously 
avoided . * 

A raw gain acore ia simply the difference between a pre- 
and a poatteat acore and reflects* the gain made between test- 
ings. It is argued that, although two groups nay have been 
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somewhat different in tenss of initial achievement levels, 
* f their expected gains would be roughly comparable after the 

same educational treatment. This would 'be true, however, 
£ only When each grpup's posttest standard deviation is the 

\ same as its pretest standard deviation, \yYere the posttest 

/ - • standard '^evi&tions aife latger than those of the pretest 

~~3ctfres, a raw gain score analysis will systematically under- 
estimate treatment effectsy Conversely, the procedure will 
/ systematically overestimate treatment effects where the stan- 

dard deviations of pretest Scores exceed those of posttest 
scores. 

A residual gain score is the difference between an 
actual posttest score and a posttest score estimate derived 

* from the combined, treatment and control group regression ^3 
* * line. Presumably the mean residual gain score for a group 

that received an effective treatment would be positive while 
that' for the control group would \be negative.* Also^ the sum 
ci the absolute values of the two j differences would provide 
.-ifvHndex of the size of the treatment effect. Unfortunately, 

be. shown alegbraically thajk a residual gain score 
analysis always underestimates the size of the treatment 

* * effect except where the groups' pretest scores are equal. 

Furthermore, the amount of underestimation- is difectly pro- * 
portional-rto the'*si2e of the iriitial difference between 
groups. 

There are other factors, suah as how t|ie treatment and 
control groups were formed, wftidh determine the appropriate 
9 adjustment procedure to compensate for \hei* initial differ- 
ences. The issues involved, .however , are very complex. Ex- 
pert advice should be sought. / ^ 
* * I 

Hazard 10 f Constructing a Matched Control Group 

after the Treatment Group H^s Been Selected 
___ j 

Finding, "matches" for treatment participants in some 
other group is a fundamentally unsound practiced Unless they 
and the treatment pu]H.ls are/ equally representative of the 
groups from which they are drawn, statistical regression will 
act differentially on the po groups and artifitially inflate 
the apparent gains of one /group with* respect to the other. 
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; Jb the most common situation, the group (s) f^om which 
jthe matching control pupil* ire drawn will be higher *achi^^ 
ing ttian those from which, the treatment group pupils are^J- 
lecteif.. Consequently, th^f control group pupils will be far- 
ther Below the mean of the group (s) to which they belong 
than ^rhe treatment group /children* On retesting they will 
thus^show greater statistical regression and their posttest 
scores will be too high , to serve as a nb-treatment expection 
for the treatment group/ participants* 

<'.■'■-- \ 1 • '>> ■ 

j; Tht correct *oce4ure for establishing matched control 
groups is to do th* matching first and, then assign members 
of iach pair randomly to the trtatnent'or the control group* 
That' ia> a large group of students, all eligible to be in 
the project, must be available* The first step is to divide 
' the group into matched pairs based on test scores, ethnic 
background, sex, etc.* so that the two metiers of** each pair 
are as similar as possible. Then, after the matching process 
isrcoaplete, some random procedure such as flipping a coin 
should be used to decide which member of each pair goes into 
the treatment jnd which into ti*e control' group. 

, Since this procedure will rarely, if ever, be .possible 
in-' real-world sittUtions, the only way to avoid thi* hazard ' 
is' to avoid matchjiftg ?( ; A better practice is to work with un- 
matched 'and therefore, slightly different groups and tot cor- 
rect for between-group differences with appropriate statis- 
tical adjustments, such as some form of covariance analysis. 
.Again, practitioners should seek the help of experts* 

Hazard 11: The Careless- Collection of Data 

Testipg and other; types of data collection must be ac- 
complished with scrupulous attention to detail* • For most 
evaluation models, the primary requirement is that treatment 
and control or comparison groups be treated in exactly the ' 
same way* JUrior variations from the procedures described 
by the ^ test publisher, are permissible in control group models 
and in certain' quasi-experimental designs* It is essential, - 
however \ that" treatment and comparison groups be tested In 
exactly the same; manner. In norm-referenced evaluations, % 
treatment group students should be tested in the same way as 
the students in the norm group* This requirement means that 
procedures outlined b^ the test publisher 'warn* be followed 
precisely* 
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Problems arise if tests are administered or scored, in 
an vinconsistent and careless manner. If there ate differ- 
ences in the ways in which* the test -takers and the norm, group 
students are tested or if there are differences in the proce- 
dures, conditions, and scoring 0 at pretest and posttes't „ticms , 
then it is impossible for the resulting^data to be accurately 
interpreted. There are no statistical manipulations that can 
compensate for mistakes made in admiijistering or scoring a 
test. - 



To avoid this hazard, the following steps should be 



taken: 



1. Test procedures must.be orderly aijd accurate 
if scores are to be meaningful. 

2. 'Test administration and scoring procedures must 

be exactly the same for ithe {treatment group as for 
the control, comparison/ br norm group used to 
generate the no-treatment expectation. Testing 
treatment group pupils ijn exactly the same way as 
p\ipils in the norming sample means following the 
test publisher's directions in every detail. 

i . 

.3. The procedures conditions, and scoring methods 
used during post testing jmust be ekactly the same 
aa those used during protesting. 



Hazard 12; The Use of Different 
Pretesting and Posttesting 



i 

1 ns t r uraen ts for 



In the norm-ref erenced^jj^pign, it is not advisable to 
change tests between pre- and posttesting because there is 
ho adequate way to. compare pretext scores on one test with 
posttest scores on a completely {different test* Since e^ch 
test publisher follows slightly 'different norming practices, 
it is likely that one test's norins will be slightly "Easier" 
than another's. These differences do not matter if the same 
test is used both pre' and post b'ut coulct magnify or obscure 
real' gains if changes were made. 1 While it is not essential 
to use the same form and level of an achievement test pre 
and post, this practice is also recommended. - 
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Some tests have been developed' so that the lover levels 
are Intended for use at the end of one grade and the begin- 
ning of the next . * In these instances, to use the same form 
and level of test for fall pretesting and spring posttesting, 
it will be necessary either to pretest or posttest out of 
level. In some grades where spring-to-spring or fail-to-fall 
evaluations. are conducted, it may be necessary to change test 
levels in order to avoid ceiling or floor effects; unfortun- 
ately, this practice will introduce an unknown amount of error 
into the measure of gain. 



Hazard 13 1 The Use of Inappropriate Formulas to Generate - 
No-Treatment Expectations 

Many projects, use an ^unrealistic theoretical mo tie 11 or 
formula to calculate "expected" posttest score? from^QjT 
other pretest scores. If students do better than the cal- 
culated expectation, the project is considered a success. 

C • 

Many methods have been devised fof calculating per- 
formance- level expectations that r£st on untenable assump- 
tions. Neither IQ scores nor grade-equivalent scores shQuld 
be used to generate no-treatment expectations. ^ For example, 
a student who has gained. ?7 year per year, on the average ; 
. since beginning school, is -presumed to continue at the same 
rate , unless a special program increases his rate. Unfortun- 
ately, grade-equivalent, gains measured from fall to spring 
.will usually exceed this rarie— even for typical Title I chil- 
dren—and treatments "will appear to be more effective than 
they really are* • M 

In norm^referenced models, this hazard may be avoided 
by generating* the no-treatment expectations solely from em- 
pirical percentile norms tables. When. control groups are * 
.used, the actual posttest scores of these groups provide "the. 
proper /^asis for evaluating treatment effects.* 



Hazard 14: Mistaken Attribution of Causality 

k Observed gains may have resulted from the Title I treat- 
ment, but 'there are always plausible alternative explanations. 

>e v plausibility of these alternative explanations^ should be 
^refully examined* before evaluation results are attributed 

. \ 
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Introduction ' 

This paper is in two parts. The first section will deal with the, use of 
\ Photography in an actual evaluation study. , It will be a loosely stated case- 
study of how one evaluator was able to utilize the camera in",several related 
but distinct ways. This section is written after the" fact - with the evaluator 
, reflecting ^pon-his use of the camera during~a series of on-site evaluation 

visits.- Many of the thoughts in this reflection have been .influenced by Susan 
Sontag's book On Photography and the author owes a great debt to Sontag for 
articulating some of the questions and issues that are related to this paper ' 
- and which should have an impact on the us'e of 'the camera as an evaluation or 
research tool. " ' 

The seWsection of this paper will-di*cus 5 some ojF the broader issues/ 
that^are raised with regard to the use of the clmera as a" research tool. .These 
issues will focus on some .of the possible limitations and advantages that should 
be considered when using the camera as an evaluation tool and a reporting medium. 



Overview 



In 1976, a team of evaluators from Syracuse University's Center for Instruc- 
f -tiorial Development, attempted to meaningfully describe and portray the workings ' 
. and characteristics, of a National "Science Foundation funded curriculum known as 
ESSENCE or Environmental Studies. This curriculum presented a unique challenge 
to the evaluation effort because it claimed to be free of specified behavioral 
-<? objectives and even went so ftp m to state that "Evaluation raises hell with . 
trust." This challenge was heightened because", ESSENCE also claimed to be con- 
tent-free. Its goal, if we can sav. that it had one, was to introduce children 
to the process of inquiry; content, was ^viewed^as a vehicle for this process. 

After -reviewing the ESSENCE materials-,, it was felt that the best" way. to 
approach ESSENCE would be portrayal - the telling of the story of ESSENCE. |e ■ 
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hoped this would provide the client - jhe Director of the Environmental Studies 
. Institute at ^Syracuse University - with a meaningful understanding- of the cur- " 
ricuftum as it was/actually used in the schools. Therefore,' the aim of .our in- - 
quiry was. primarily description. - We .attempted to make no claims of merit £e? 
■ se. Rather we hope! to capture tbe^ various characteristics and activities of 
ESSENCE as we discovered them so } that those who were not present when these actions 
were taking place might still be able to s hare our appreciation of some of the/r 
meanings. This would enable the^client to come to his-own conclusions and make S 
his own judgments. , * 

We decided early in our planning that we would use a camera as one of several 
fria -ts fo^tthe site-visits in* order to capture and portray what ESSENCE, was 
- in real classrooms, with real teachers and real students". The study cbnsisted 
of three ma jof components:' site-visits to schools where ESSENCE was or had been 
used; a separate review of the literature on affective" education that we hoped' 
would link our observations with the Research literature; an extensive ques- 
• tionnaire sent to-participating teachers and administrators to be responde 
in writing or on an enclosed cassette 'cartridge. The 6amere was used in 
visits. - " * * . 

The notion of using^a camera as one of our research tools was reinforced 
by some of the ESSENCE activities .'which Se were probing. One of the more -highly 
touted activities of the ESSENCE -curriculum wis Wuse oT a Polaroid camera by 
the students as a tool of inquiry. The students were given Polaroid cameras 
with the assignment, of "take a pictured something invisible," or "take a pic- 
ture i hat is some positive evidence that something natural happened," or "go ' 
outside and take a picture of power," or "take a doWe exposure- that portrays 
some change, « ESSENCE purported to 'be a process of inquiry, and th^mer^ was, 

sro 
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viewed as instrumental in this process. It focused student attention on a ' 
specific problem or issue.. It framed specific actions to the exclusion of 
others. It recorded' how things change and/or remain the same. 

Because the evaluation team was 'uncertain as to what to expect- while con- 
ducting our study of ESSENCE we adopted a certain ESSENCE point of view. This 
point of view stressed process rather than specified outcome, openness to what- 
ever, or wherever the study" lead, and a. willingness to deal with a certain amount 
of uncertainty, as our inquiry 'unfolded. Therefore, we decided to use the camera as 

♦ 

an instrument for focusing on specific issues, and as -a means of visually recol- , 
lecting some of those myriad events and activities that were witnessed during 
a hectic three wefeks of site-visits. And, most importantly^ we hoped thrt what ♦ 
was captured on^film would be useful 'to the client in better understanding some 
of the .very complex and unique characteristics of ESSENCE and the people who 
utiiieed i't. ' *"* ' % 



Three- Uses of Photograp hy in the Portrayal of a Science Curric ulum 



Because our knowledge of what ESSENCE comprised was very vague', the camera 

initially functioned as a wans of framing and isolating certain actions and events. 

* i 
The use of*the camera as a means of focusing upon one specific .event or activity 

during the site-visits was a value-laden act' in itself. Implied inW photo- 
graphs of the site-visits is the assumption that what was photographed was some- 
how worth considering as' a small but isolat^portion of our portrayal study* The 
camera forced the site-visitor to, at least /momentarily, focus his attention on 
a specific activity. In what could loosely be described as a goal^free evaluation 
the camera became an instrument of goal-seek^ng^helpin'g the sitelvr&jjwr search 

* Stephen Kemmis speaks to this issue briefly in "Telling it Like it is: Tne • 
Problem of Making a Portrayal of an Educational Program, January 1974, University 
of Illinois, Champaign-Urbana, (mimepgraph). ., ♦ 
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out patterns and look for order in the seemingly unordered activities going 
on continually around him. From the vei3§r beginning, the site-visitor was com- 
piling a series of individual .images - both mentally and photographically which 
would, much later in the study, be pieced together to tell the ktory of ESSENCE . 
Just as. the Polaroid .cameras were used by the students to focus their questipns, 
their inquiry, the site- visitor utilized his camfera (Olythpus 35P.CJ to bring order 
and clarity of value to hL'; study. This. was especially .important in tn£ early 
stages of the study when we were neyer certain what to expect from ESSENCE or 
oven what questions to ask about it. 

This ordering was vital to our* study because we understood from the be- 
s *nving ti,at ESSENCE, because it concerned itself more with process than product, 
could mean many, things to different people. Our objective was to portray, in the 
most meaningful way possible, "A Story, of ESSENCE." The emphasis in this title 
must be put on tfie article "A" - because we were convinced that what we were 
portraying was but one of a number of stories that could be written about this 
National Science Foundation curriculum pjroject. The camera was one way events 

s 

and activities were, singled out and isolated and then pieced together to re- 
create one of many possible stories that would accurately and adequately/ portray 
ESSENCE. v * 

The camera was also used as a primary mnemonic device, along with a portable^ 

» * 
tape-recorder; to aid in the site-: visitor's later recollections of what he ob- 

(served during hi£ visits. This took some of^the burden off the evaluator in terns 

of trying to keep track of the many happenings that he experienced during his 

visits. The camera and the tape-recorder almost completely replaced ncpte-taking 

'in the site-visits, thus leaving the site- visitor free to more readily "grasp • 

the insider. 1 s view of -the program" (Kernmis, 197U): asking* probing questions of 
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teachers and learners; watching for subtle interactions between teachers ancf* 

children as well as between childretf and children; and trying to understand fhe 

total curriculum without fear of missing some of its more Specific components. 

— There is a sense of professional well-being in knowing that one is collecting 
i ( — - . 
as much usable ^Ldence as possible in any ^valuation study. This was particularly 

true in the site^fesito^'s "observations of ESSENCE in action. Not knowing before- 
hand what'to. expert, it was comforting to know that dat.a (in theform v of taped . 
conversations and photographs) were fcfeing accumulated, and that these* data would 

be helpful, in fact necessary, in piecing together a story, an adequate portrayal 

i 

of -this curriculum. Like the atomic physicist trying to capture' images' p 5 sub- 

/ * • , ' # 

atomic particles, we could only speculate e about wjiat we would find - but were 

confident that if there were something to capture on film, we would discover it. 

And like the physicist, we were very aware that we might~not« fully understand 

whet it was we were observing until well after the study's on-site observations 

were completed^ The photographs taken during the visits to the schools were a 

form °f insurance that guaranteed that tfce site-visitor would have somethihg to 
* * 

say about what he "saw" during his travels. The camera was his guarantee that 
he would not forget the obvious - that those small details that give meaning 
to the total picture wpuld not be omitted. 

Although the^ camera was utilized both as a means of focusing on* specific ac- 
tion and as a memory back-up device, we felt from the beginning of the study that 
good photpgraphs of the curriculum in practice would enhance ,bur/study of ESSENCE. 
We hoped to make our report as descriptive as possibife and we did not want to rely * 
solely on words, to paraphrase the photographer Lewis Hine^ if we felt we could 

m * 

.tell the story in words alone, .,we wouldn't have needed to lug a camera around (Son- 
tag, 1977). The ESSENCE study i? a story of words'. It is a story about people^ - 
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young and old, together engaging ijLthe activity of teaching and learning* 
"The Story of ESSENCE" relies 'heavily on N the real words of real people - but 
where' it^was possible, photographs were used to illustrate the objects of 
conversation and to heighten the worded" story . Thus, the camera was a most, 
helpful tool in helping us repont pur findings to the client in a way that 

both factually informative and meariingful* Although th£ photographs by them- - 
selves are only isolated images, mere bits 'and pieces of evidence, when they 

• • • . . • - / ; . 

were paired with certain conversation? or usred to illustrate particular circumstan- 
ces, we believe they added depth, and human feeling' to the report/. The$e* were 

Instances when it was felt that the photographs led to a better understands 
, jT £ conversation that was taking place or a unique activity that yas happening. 
When tTie~£aped conversations and photographs are used together they take 

\ 

on a reality themselves, a reality that may^fee greater th&n the sum of " each taken^ 
separately. This reality is basically independent of the recorder even though 
it is his point of view that informs $*at~ questions are to be asked and what 4 seg- 

. V * 

* ' - r • 

ments of visual reality are to be captured on film. This jnarriage of photography N 
and taped conversation gives the client a ,sense of autonomy in h£s judgments and 
decisions. He becomes a vicarious, participant in the events that have taken place 
and is freed from his reliance on the verbal interpretation o^the event though 
a mediating voice. 'The use of photographs especially in portrayal- type evaluations 
allows the client to comS.tOsmore informed conclusions, to make * his own decisions 
about the wo^th of a program, curriculum,, or other activity. .Without dealing 
with the question of 4 the subjectivity or objectivity of the photography in evaluation 
research, the. camera can help the client' in better verifying his own opinions about 
the findings and recommendations of an evaluation report. Photographs are ad- 
ditional types of "evalu^££&n evidence, and if the^ are riot used surreptitiously 
and defemed admissable within the particular context- of the evaluation study they 

gH 
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can add a sense of visual validity to more traditional modes of evaluation. 

Issues and Problems Surrounding the Use bf Photography as an Evaluation Instrument 
* In the 4 previous section the usefulness of the ca^erajSjg^ an evaluation tool 
. was briefly, jDut positively discussed. This • raises some questions as to how dif- * 

ferent the camera is from other evaluation tools - for example statistical analysis. 

*\ ' % + / // 

In fact, statistical analysis and the use of photographs m evaluation research 

* /f J* » * 

may be analogous - bbthJFocus upon one event or activity^^to^thje e&cfusion of others; 

an 5 both ha\fc a quasi-mystical fascinat ion~for some people who see both as some 

s^hoiic representation of "Truth" or reality. 

T fact that photography is but a tool in the evaluation process can hot be 

ir.derplayed . It is instrumental in achieving other ends (description, determina- 

tion of worth, reasonable decisions) and is not an etad in, itself. On* uses a 

.shovel to dig a holev'and iihless-one is a craftsman'or tool designer, the type 
* j . * 

and design of the shovel are relevant only in so far as, they fit the needs of the 
job °ta be done. Th.e camera will sometimes fit the needs -of the job to be done, 
sometimes it won't. We have been warned by Robert Stake that evalu^tors "should 
choos'e methods to fit the issues, even if forced then to employ weaker and less 

respectable methods" (Stake, 1977). The evaluation team who compiled the data 

• « / 

for "A Story of ESSENCE" and the^ client for whom it was 'done, felli 'that .in this ' 

- . : ' v ' . . * ./ 

case the usfe^of the camera was a "respectable method." However, in another situsK 

a * t * 

• tion, at another time, another method, a riiffer^ent tool might.be more appropriate. 
The f*'.t the carr.era excl Aht scr.e -ctiviTi^s iz ~r.e ir.cl-sion^f otners 



is *not a positive pr negative aspect in itself. Rather, it is how we 4eal with \ 
this limitation that makes' it relevant;*,^ evaluation researc^ 1 ^^ site-vi^ito*^-^ 
of the ESSENCE study believed that the 'act of focusing on some events and excluding » 



, . others was a valuable ehdeavor in bringing ah initial sense of" order to. a seemingly 
disordered, array of events.' The behefit of focused 'specificity in .this case*was 4 , 

ERIC • 123 



. -8- .... 

seen as a necessary beginning^to the formulation of an adequate overview<~of 

the curriculum. "The problem at issue here is how much specific focus is enough 

to pick out i:he important components of a program and how much is too much r so 

that the total overview is' lost in obscure detail. Again it is Robert* Stake 

/ / f • 

who best articulates this question "What is more important: to tell of some very 

special things about the program or to provide the most veridical portrayal of 

1 * 
the program?" (Stake, 1972). Stake advocates the latter. Is it possible that * 

the prudent use of photography. as an evaluation tool might enable evaluators to. 

-j both - tell some special things about a program without detracting from ^ the ( 

</erail portrayal of that program? It is not so much that "special things about 

^' z^ogi m v necessarily detract from its, "most veridical portrayal" rather 

r 

; >,e perseveration of the evaiuator with these "'special things" that cloud 

and distort (meaningful program portrayal. This can be said about any evaluation 

/ ' ^ 

method of topi that is seen as an end in itself rather than an instrument to 

another more informative, value-based end. 

It is this infatuation with the tools of the evaluation trade rather than 

with process of valuing itself, that must be avoided. Th^fc^uera does not seem 0 

to offer any great panacea to th^problems of evaluation researbl^y It does offer 

a novel point of view to the' field, mat just as the limitations of\a complex 

statistical analysis^or research design^must be kept up-front in evaluation 

research - so too the limitations of the camera must be fully explored afrd 'dealt 

* 

->with. * 

Because* the number'of photographs that can be tS^egp- is virtually unlimited, 
pkotogr^phy^j^inforces a segmented view of reality consisting of small, separate 
ypits of an apparently infinite number. This makes the world more manageable be- 
cause the photognapher or the viewer deals with only one aspect of reality - one 
separate photograph -'at a tim^CSontag, 1977). It was this manageable nature 
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of photography which aided the ESSENCE site-visitor - bringing order to the 
initial stages of the study. 

The individual photographs only showed how sfparate and distinct parts of J 
the curriculum* looked . They did not showChow it functioned (Sontag, .1977). How 
something functions implies action and time is a necessary condition of action. 
Indif*4uai photographs of "ESSENCE did not, by themselves, have the power to mean- 
ingfully c6nvey the workings of the fctifrriculum over time, even when .the photo- 
graphs were viewed as a group. • It was the* narrative quality of the taped cbftver-' 
sations which placed the activities of ESSENCE (and the photographs) within the 
context of time. Narrative deals with time and action over. time. It is only 
within the context of time that we can talk meaningfully of .process. An evaluation 
study" that stresses process mustTeal with action o^eF^ime. Narrative, therefore, 
befcomes a necessary condition of any process-oriented evaluation. 

Photography and* the discriminate «use of the camera have a great potential as 
> useful and informative evaluation .tools. The evaluation profession must make Cer- 
tain that these tools are not elevated\to ends- in themselves - but rather that 



they be seen, and remain to be seen {as only tools in the growing repertoire of 
the evaluator. Photography can aid the evaluator in focusing inquiry and collecting 
data. But the ends of evaluation, determination of merit or worth, the making 
of rational decisions, and the describing of events and activities must not be 
prempted by the tools which serve those ends. 
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< ' ^ 

Thia article will describe an evaluative "process derived from the - 
suthors 1 experiencea in meeting the mandated evaluation/documentation 
raqulrementa of a federal project feigned to foater educational change 
and improvement. Thia article conceptualizes evaluation as obtaining 
information vtiliaable at two le/els. Evaluation ia seen as a mean* for 

empowering project ataff to he able to monitor progress towards meeting 

! ' • 

project goals, yielding information useful in the process of designing 

/• ' 

and redesigning techniques for advancing to theae goala. In~addition, ° 
evaluation la aeen aa yielding enabling information for the drawing of 
generaliaabla conclusions about the change process. 

• • • * • 

The federal mandate for documentation led to a number of Issues that 
needed resolution. In ord« to acquire wide-ranging information about 

issues bearing. on project outcomes, such as differing perceptions of the 

a? 

political forcefield, atratagiaa used by project staff to cope with the 
forcefield, and the interim and ultimata goala of the project staff, the 



active^ cooperation of all staff members in the keeping of records was 
necessary. Thia meant that ataff members needed expertise in record 
keeping end motivation to keep theae records. Realistically, nany people 
are not, trained to keep accurate' records and need support in developing 
their own evaluative akilla. The problem of motivating harraeaed ataff 

t 

to keep records is even more difficult. It is partially Tolved by con- 
cerned aupport from the evaluator and by making information kept for 
documentation purposes a vital component of individual and overall pro- 
ject management. ' ^ 
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Evaluation is seen as a reciprocal process between the equator 
and the people he deals with. The model assumes that staff must be 
empowered with a proactive role in which they have an impact both on 
the instruments that they use to collect data and the techniques used * 
to analyze that data. The evaluator is obligated to portray to the 
people he collects information from the emergent themes he finds in tjie 
data. ^ They, in turn, have the right both for instruction in interpreting 
these \hemes and an opportunity for affirmation or denial of them based 
on their personal perceptual framWork of the validity of the evaluator' s 
findings. 

Bqvond the issues of accurate, comprehensive documentation lies the 
problem of theoretical context, an issue not customarily" dealt with in 
valuation. The authors feel that the generalizable information aVail- 
able from a multi-faceted, multi-data source, case study has been under- - 
rated. /There are some ^theoretical models which provide insights into- 
the delineation of the change process. The current federal mandate for 
docu^ntation/evaluation can ptjvlde information of a scope that can 
test a theoretical, explanatory framework so that generalizable informa- 
tion about the^piiange process <;an be obtained. * - 

This paper will describe four aspects^of the facilitative evaluation 
process. The first section will discuss how the evaluator goes about pro- 
ducing the evaluation design while 'establishing a good working relation- 
ship with practitioners. Next, a description of the steps taken by the 
evaluator and practitioner for implementing the evaluation design are 
described. Then, some examples of theoretical contexts in which to con- 
sider the change process will be discussed. Finally, some- suggestions will 
be sketched as tJ^techniqu^B for analyzing the data. 
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/ Designing the Evaluation 

The following seven events describe how the evaluator initiates ^ 

f ' / _____ 

and continues his work: 1 —r— 

< S 

N Event 1: Interview between Evaluator and Practitioner to: 

(a) establish trust level 

(b) discuss practitioner's role 

(c) assess level of evaluative sophistication 
\ of practitioner / 

(d) determine practitioner's perceived needs* 
for information 

(e) determine practitioner's ideas for collecting 
information » 

Event 2: Compare results of total* interviews with the overall 
project design, analyzing discrepancies and exploring 
I causality. 

Svent 3: Evaluator derives overlll project design, given con- 
siderations of: 

(a) project goals and objectives 

(b) practitioner's roles 

' (c) practitioner's ideas for collecting information 

and - • # 

( (d) level of evaluative sophistication of practitioners 

Event 4: Handcraft an individualized instrument (a) for each prac- 
titioner based on overall evaluation design and practi- 
tioner's level of sophistication. 

Event 5: Check instrument (s) Mth practitioner to determine if it 
fits his needs and is usable. Instruct the practitioner 
in utilizing the instbttgnt. Redesign instrument (s) if 
necessary. 

Event 6: Implement work process using individualized instruments 
to collect needed information. 

^ Event 7: Conduct follow-up interviews to: 
,i (a) monitor progress 

(b) update instruments 

(c) describe emergent themes and check with 
practitioner's perceptions. 

The purpose for using the seven event model is to make it possible 

to collect data that is "accurate information" which will be utilized 

cooperatively by staff and evaluator in planning project development. 
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The Facilitative Evaluation model is intended to encourage growth both • 
in the competencies of individual staff members and in the achievements . 
of the project as a whole. In this model, the evaluator' s role involves 
gaining the trust of the staff, involving staff in the design of instru-^^-^ 
ments to obtain" information and keep records, educating staff in evalu- 
ation techniques as they participate in the evaluation/planning process, 

t 

and participating with staff members in utilizing instruments. These 
events empower staff with evaluative techniques so that they can, use , 
aata for their own needs as well as the general project needs. 

4 

This model uses an interview technique, Event One, to gather infor- 
..^oa\u! to facilitate staff cooperation! Primarily, one-to-otje in- ' 
terviews with staff and other appropriate people are scheduled. The ' 
interview process, when done well, closely parallels a' helping-counseling ^ 
process. There are many factors that contribute to successful use, of 
this technique. The evaluator must assume that the person being inter- 
viewed is- doing important work and is worthy of consideration as » pro- 

— t 
fessional. Both parties should be seen as equals, discussing a situation 

of mutual interest and concern. This attitude will convey, itself in attest 

_ - - J 

tive listening, positive non-verbal encouragement and appropriate responses, 

with. the evaluator putting himself into a relaxed state with focused atten- 
tion upon *the person. A casual environment such as a informal lunch or dis-*- 

* * * * 

cussion over a cup of coffee provides the low-key unaggressive, non-judg- 
mental atmosphere that is most productive. If the process is going well, 
a synergistic phenomenon should take place with two minds becoming more 



productive and energizing than one^ Work is important and people aria* 
usually eager to discuss and reflect on it. This synergism is so re- 
warding and productive that experience has shown 'that staff .members 
often strive to duplicate these conditions with other colleagues, 
increasing the creative power of the totAl project. 

It is important in the 'interview prqcess to determine the.pracm- 
t loner's current capability to evaluate information and his ideas on 
how to collect it. Of necessity, when documentation is important, • 
staff must keep records. However, it is futile to expect them to \eep 
ecords they are unable or unwilling to* maintain. 

Event Two describes the process of comparing what people actually 
do based on interviews and observations with the overall written project 
design. It is well known thafc roles specified in proposals or planning 
documents often radically change both because of the people filling thera ■ 
and because of the evolving needs of a project. The evalyator needs to 
make sure that staff "realize that thiols a common occurrence that does 
not imply blame. What is Important is that some sort of record be kept 
concerning the context in which the change happened so that knowledge 
can be derived from tMs phenomenon. Event Three, the creation of the 
overall 'evaluation design, is self-explanatory, ^lbeit the difficult 
crux of the matter. Event Four describes the selection or building of 
* individualized instruments an£ documentation systems based on a realistic 
assessment of the practitioner's interests, and capabilities in conjunctio 

0 

' with the overall evaluation design. 
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Event Pive describes a critical step in the evaluation/documen- 
tation process. It Is important to 4heck back with the practitioner 
to' see if he can utilize the instruments designed for his use. The 
practitioner's capabilities as well as pragmatic considerations of 
available time and resources may have been over or underestimated. 

It* is often necessary to work through an instrument with the practi- 

. , \\ 

t loner using a real life situation as an^ example so that the practi- 
.oner may get realistic practice in using it. Finally, instruments 
▼ nave to be redesigned. It is better to redesign an instrument's© 



A 

by \^ 



, tv.cclcioner will' record data than to lose the information 
an instrument that is too complex to be accurately filled out 
> by the evaluator. 

Event Six sees the'practitioner using the newly designed instruments 
to collect iafermatioft. His role in this capacity is that of a partici- 
pant observer. Event ^Seven describes the conduction of follow-up inter- 
views to monitor progress and update the individually designed instruments 
It is important that the evaluator show sustained interest in the data 
jthe practitioner gathers. It is not always convenient or agreeable to 
, record information. The evaluator *s personal concern with the data/SttS 

* s 

the practitioner as well as the practical utilization of the data in the 
project will help to; alleviate this natural phenomenon. In addition, 
the evaluator needs to monitor changing project goals and practitioner 
roles and update his instruments accordingly. Then, too, it is hoped 
that the practitioner will become more sophisticated in utilizing evalu- 
ation as he gain^ practice in the evaluation process and adjustments in 
instrumentation Should be made accordingly. These encounters also offer 
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the evaluator the opportunity to portray to the practitioner the in- 

». • 

sights and repetitive themes that he has discovered from hisWrall ' 
^ perusal of the evaluative data and to discover whether or nqt these 

themes are validated by the perceptions of the -practitioner. ' 
In summary, these seven events describe what' the evaluator does to 

t 

procure practitioner participation in the collection of information. The 
events are conceptualized as a means to train 'practitioners so that they ' 
* . ~can serve as participant-observers in the field. They will then have the 
capacity to validate the emergent themes presented by the evaluator and 
to derive their own emergent themes for presentation to the evaluator. 
From overall project goals and, where available, emergent themes, the 
practitioner develops work goals. The planning-replanning cycle to io- 
CQ plement these work goals will be described in the next section. 

* Implementation and Util ization of Data- by Practitioner 

. In this section, facilitative evaluation will be discussed primarily 
in terms of steps that practitioners take to bring project goals into 
reality. These steps are designed to roughly replicate, in a pragmatic 
manner, the steps of .goal setting, implementation and reassessment that 
are so useful in an innovative project. 

The steps in implementation and utilization of data by practitioners 



are: 
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I* Select a work goal 

The practitioner defines clearly a goal and ' 
discusses the setting where it will occur. 
The evaluator should offer clarification and 
help in this process. 

11 * Discuss the present reali ty and' identify the pro blem 

The practitioner takes an objective look at the 

situation. The evaluator should encourage this ' 
orocess by responsive listening and questioning. 
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HI. Conceptualize, the most effective solution 

The practitioner conceptualizes the.most effec- 
tive solution(s) to the problem. The^evaluator § 
> can be\especially helpful by offering encourage- 
ment conducive to inventive, yet feasible con- 
ceptualizing of the luture. 

IV. Plot steps to the solution „ 

The practitioner £lots the steps needed to bring 
the goal from where it is presently to where the ' 
practitioner would. like it to be in th<* allotted' 
time. . / 4 : 

V. Establish checkpoints 

The practitioner establishes completion times of 
steps as, tentative checkpoints. These checkpoints 
serve to motivate the practitioner to- move ftorward. 

VI. Design evaluative indicators for each work step 
These indicators are milestones which indicate 
readiness to move to the next step. (Or a failure 
* vhicih indicates need for re-appraisal of (a) the 
• steps to aqhieve the g oad . -or (b) the appropriate- 
ness of. the goal). 

VII. Implement initial steps. of "the plan 

As each step is implemented and the evaluative in- « / 
dicator has or hasn't become reality, the practi- 
tioner' should log warning signs, that indicate the 
steps to "compile the j^Sal may need revising due to 
new- information. Conversely, data may indicate 
success.' 

VIII. Assess progress at checkpoints " . 
' This, is done %o determine whether the next steps are l 
feasible considering the cues that the practitioner 
ha3 collected. 

IX. Continue progress 

Continue to the next step or replot steps. If useful, 
devise a new, more effective solution to better accom- 
modate the new reality. 

X. Complete all steps (^discontinue goal ■ * 

Data indicates whether practitioner should continue on • ^ 

/ present course or discontinue the work goal.' 

In actuality, steps one to. three, the selection of a work goal, de- 
lineation x>f -pjrfesent reality and conceptualization of an effective solution 
to 3 problem, are not necessarily done in linear sequence. Instead, suc- 
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cessive approximations of one step reverberate on t&e findings of 

another step which in turn initiate new thinking* of an earlier step. 

The selection of a york goai-, sup.ont, is dependetit on a number. 
* , v ' - 

of factors. Among them are overall project goala, identified emergent^- 

% 

themes, staff interests and competencies and the time span of the pro- 

ject witTi the ramifications for what is feasible. Step two, an assess- 

meat of the present reality has a heavy impact on the selection of a work 

goal. The factors to be considered in such an assessment are "numerous. 

; - * . % 

Critical among these factors is an unblinking,, realistic assessment of 

\ 

political realities both in the actual system being worked on and the 

a.-.er environment. Aside from the "ftolitical realities, the socio-cul- • 
cural-physical climate of a system needs careful consideration. In the 
end, the number of factors considered *ih an assessment of current reality 
is determined by the perspicacity of the staff as well as the reflective ° 

' * 

probing of the evaluator, ^ x * 

Step three involves the conceptualization of the most effective so- 
lution (s) to the problem. Given the context of an overall work goal and 
the present reality as perceived by the people in the environment, practi- 
tioners decide what the most effective solution (s) to the problem are. 
Practitioners, aided by informed questioning by the evaluator, need, to 

r 

imagine what a better reality would look like. A discussion of the context > 
within which the problem will occur and feasible alternative solutions^ will 
help the practitioner think through the direction to take. If the problem 
is particularly complex, a brainstorming of helping and hindering factors 
maybeof assistance. During the process of /the practitioner working through 
these three steps, the evaluator can also be sdrtingout the project's per- 
ceived underlying problems, its underlying philosophical asswrprXons, „ ' 
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•checking their veracity against the perceptions of practitioners. , T,hese 
sessions offer the evaluator the chance for reflective portrayal to prac- 
titioners o s £ emergent themes that he has seen from his wqjpL+j-^' 

Once a consensus on the future objectives is reached, the actions 

t 

necessary to bridge the gap between present reality axld a fixture objective 
need. to be derived. This is a careful meticulous process which relies 
heavily on step two, i.e., a forthright unblinking assessment of present 
reality, both political, and social. * Step four requires the practitioner 
to estimate a realistic time line for the Completion of feach action of the 
vwk goal. Having established the actions £or the work goal and affixing 
tech action with a, completion date, step five continues, the process by 
establishing indicators to help the practitioner know that he has achieved 
or failed' to acjiieve the necessary actions. . « * 

Hie sixth step is to implement > the plan',* recording whether or not the 
evaluative indicators have been met. Different indicators signifying that 
an' action is completed may indicate a need to readjust subsequent' < actions. 
Sjjeps seven, eight and nine are to assess progress at checkpoints, continue 
to next step or replot steps and finally, complete^ all st;ep3 or discontinue 
* goals as data indicates. The steps are conceptualized as a circular process 
ofj^al^s.etting, implementation,, checfcingand re-appraisal. The choice of 
abandonment of an untenable position is built into the process. It 'is 
assumed that a position may become untenable because of an inability to 
achieve the intermediary steps* necessary for attainment of the goal or be- 
cause the goal itself - begins to ^>e seen as a mistake. /It is impbrtant to 
document" the events leading up to failures so' th^t*' appropriate analysis may 
be Wde of the causes of such failures. This whole series of s,t^ps is de- 
signed to^ield written records ^hafc^are of systematic use in decisign making 
■ . '. / '• ■ \s> 
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bv practitioners as well as providing participant observation field 
records for the evalutor. 



Theoretical Framework 

> "s " . • • • 

Aa^the authors stated in the introduction, this evaluation model 

> m , ' 

serves two purposes. The first purpose, discussed in the previous sec- 
tions, is to provide, data for better project management. Tfie second 

/ 

purpose is to gather generalizable data. about the change process. In 
this section a discussion of the theoretical frameworks dealing with 
the change process will reveal possible themes that may assist thfe 
^valuator in a better understanding of the proceedings that are being 
assessed. 

"Industrial managers are fond of noting that change is the only 
thing that remaifir-conatant in their lives. Yet despite the common 
occurrence of organizational change, its dynamics and underlying pro- 
cesses are understood in only rough, ill-defined ways. Managers and 
social scientists who creat* and study change situations find that or- 
ganizational changes involve multiple sets of complex variables whose 
identify, interaction and impact vary from situation to situation." 
(Barnes, 1967.) 

Because of the varied and rich data at his disposal, the project 
evaluator is in an excellent position to identify emerging thebes and 
to portray these themes in a larger theoretical framework. The authors 1 
intend to briefly describe three sociological theories that provide con- 

* • a 



itual contexts by which to consider the change process. The three 
theories are: conflict, theory ,s exchange theory and interaction theory. 
Although there are certainly other theories \hat would be -appropriate to' 
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consider in gaining a larger theoretical framework in which to consider 
>tft£ pro'cess of change, it is not the authors' intention to present an 
.exhaustive study of theoretical models related to change but merely to 
spark interest in* the notion of incorporating theoretical' thinking into 

the evaluation process. t 

Conflict is a theme that recurs 'frequently in the change process. 
y ds Coser (1956),. in his study of conflict, cites a number/of scholars 
• , 3 ^-nmel, Sorel and Marx who maintain, that confliat^is a binding 
. t well as a disruptive one. Heroes on to state that interest 
groups will form and reform around issues that are conflict laden. • 
"...when a social structure is no longer considered legitimate, in- 
dividuals with similar objective positions will come, through conflict, 
to constitute themselves into, self-conscious groups With common interests." 
(Coser, 1956.). Most theorists in talking about conflict tend to agree 
that conflict cannot be repressed, flow it is expressed, however Jis re- 
flected in the characteristics of the individuals or groups dealinVwith 
it. Coser states that there is a difference In group behavior when con- 
flict exists within a group and .when it is directed at a group from an . 
external source. Groups who have very close internal, relationships tend 
tq repress internal conflict longer, making, the conflict much mote intense 
when it finally occurs. Groups that are being attacked from without tend 
to demand more commitment' from their members with fewer conflict issues 
allowed to surface from within.' w 
' s A final element in conflict theory that the authors will consider is 
. Coser' s definition of. realistic and non-realistic conflict. He states that 
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while realistic conflict involves resolving conflict focusing- on the 
actual issues, non-realistic conflicts axe based on real issues, but 
deal with them in an indirect manner, resulting in a. conflict resolu- 
tion dealing with false issues. Thereforei there is not a true reso- 
lution of the -actual issues. Noting that conflict theory concerns it- 
self with £jje behaviors of individuals and groups engaged in con- 
flict situations, the authors feel that a look at exchange theory would 
fi*x»nd the reader's theoretical perspective of human behavior, around 
lss.ues of change by examining and analyzing other factors that affect 
"-h*r people do* In this theory, Thibault & Kelley (1967) 'are concerned 
with interpersonal relations *and group functioning or put another way, how 
people's behaviors are influenced by the costs or rewards that they per- 
celve they will experience as a result of their actions. During the pro- 
cess of deciding what action to take', a person is continually weighing the 

consequences of -his behavior, -often with opposing notions about what his -* 

*• • • 

actions- should be. Thlba\ilt & Kelley (1967) define a variety* of explanatory 

terms that help decide what action a person will take. Norms, or those 

agreements about what behaviors are and aren 1 tr . acceptable that carry with" 

thea a social process to enforce compliance, afford a certain degree of 

predictability to the actions people may take. Status is also important 

when deciding who will do what. The positive status of a person influences. 

k others to more frequently behave as he does, giving him more power. Power. 

and dependence are interrelated phenomenon. An individual obtains power 

by others being dependent on, him. His power is greatest when he can operate 

as he wishes without being concerned with others actions interfering. / 
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Exchange theory takes into account all the possible behaviors a 
person may exhibit in any one instance and analyzes the consequences 
each of the behaviors will have for him. While exchange theory deals 
largely with people's behaviors, interaction theory deals with the 
perceptual events in people's minds. As change in an organization in- 
volves constant hUman interaction, it is usef ul ,to look at some concep- 
tualizations of how and why humans interact with each other as they" do. 
Interaction theory looks at human conducted how people function with 
other individuals or groups. 

George Herbert Mead and Herbert Blumer (1969) identify two kinds 
of interaction in human society. Blumer calls them "non-symbolic" -and 
"symbolic" interaction. Non-symbolic interaction takes place when a 
person responds indirectly to the actions of another without interpreting 
them. Symbolic interaction assumes that a person takes meaning from 
others' actions and responds by behaving in accordance to the meaning 
. he has affixed to the others' actions*. (Blumer,. 1969) . Blumer also 
uses r th* term "Joint action." This is characterized by actions ranging 
trom the interaction between two people to the complex social workings 
If an organization. These "Joint actions" make up society, according 
to interactionist thought. "Each participant necessarily occupies a 
different position, acts from that position, and engages in a separate 
and distinctive act. i It is the fitting together of these acts that * 
constitutes Joint action." (Blumer, 1969). People perform tasks by 
assessing the Job to be done, looking at their own actions and the actions 
of others and fitting these acWons together to accomplish the task. In- 
teractionists believe that people's actions are determined by the way -that 
they perceive reality.' A person's perception of reality and the role that 
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he plays in that picture of reality, therefore, effect the alliances 
he develops and the actions that he takes to maintain or change these 

alliances. - t 

f \ in summary, the authors have briefly described parts of three 
sociological theories thatSiave implications for the change process. 
Our description of conflict theory looked at how conflict affects 
people in their dealings with others. The distinction between realis- 
tic and non-realistic conflict was also described. 

Exchange theory deals with people's behavior and how these be- 
haviors are influenced by the* costs and rewards that "people perceive 
they will experience as a result of their actions. Norms, , statua/ 
power and dependence were all discussed in relation to this liheory. 

Finally interaction theory was. discussed. 'Interaction theory states 
that people's actions are determined by their 1 perception of .reality * and 
the role that they play in that reality. Symbolic and non-symbolic inter-, 
action and joint action were defined. 

' Each theory gives the evaluator a different but not unrelated perspec- 
tive of the change process. A conceptualization of ^the^evaluators role in 

« 

portraying this knowledge for the practitioner might be, seen as Millte et 
al.' have portrayed the change prqcess in their theory of social reform 
(1977). First, an identification of the actors in th£ processes made 
and then an identification of. the variables which describe, the process 

in which the actors are involved is determined. An analysis of the inter- 

< 

relationships that exist between the actors and the prodess that they are 
involved in .characterize 'the change process. With this conceptual model 
as a base, "the usefulness of a theoretical framework becomes evident. * 
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Conclusion 

Evaluation* has been customarily conceived of as formative and 
summative in terms of success and failure. Given the time and money 



limitations Of much evaluation, this is indeed,. all that can be hoped 
for in many cases. However, given the scope of information that can be 
collected using techniques of evaluation/documentation, the authors 
would- like to briefly sketch some techniques of synthesizing this in- 
formation into a comprehensive whole. f 

As the evaluator follows the techniques outlined in previous sec- 
tions, he is faced with reams of data collected both from his vantage 
point and from the vantage point of the participant-observers he has 
trained. He is confronted -with the task of placing differing world 
views into a portrayal of the whole. The authors concur with the 
techniques suggested by Carini, Engel & Hein (1978) fbr the application 
of qualitative methods to program* evaluatidn. „ As information is col- 
lected, the evaluator should immerse himself in the contemplation of 
it. Themes should eventually emerge from the contemplation of this 
collection of ^ata. The evaluator should then begin combining this 
data into a loose we*ve of meaning* Once this is ddhe, he can commence 
cross-ma trixing theie emergent themes preliminary, to an analysxs or the 
data to determine the relative influence of various actors and their aa- 
tions* on the process. The data can then be cross-checked with theoretical 
premises.' .The evaluator, because he has access to all the rich array of 
inforxftation is in an excellent position to portray data in a broader, 
theoretical framework. 

- The authors" feel tKat' the interaction of practitioners~with~l:he 
evaluator is crucial throughout this process. Peopled perceptions of 
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their experiences trigger theoretical understandings which when 
revealed to the practitioner empower him with new insight.. Besides 
utilizing the perceptions of a number of participant-observers,^ with 
, i differing world. views in his* 'immersion in the data, the evaluator is 

o&Hgated to continuously check and cross-check with participants both 
. the emergent themes that he perceives as well as the ones that they per- 
ceive. The evaluator needs to make fragmented data whole and mirror 
— it back to practitioners making it a comprehensible entity for them. 
Emerging themes are portrayed to managers so that they can change 
their procedures, if necessary. Data is thought of as representing a 
phenomenologically rich world in the process of. becoming. 

The authors feel that, -while human beings are amazingly complex 
organisms, there is still much that is universal from human situation 
to human situation. They suggest consideration of this'model to gain ^ 
, information that will be useful in the "practical art of getting things 
done" (Stake, 1978, paraphrase) while allowing for the chance of relating 
case study data to a larger theoretical framework. 
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Highlights In Thisjssue 

What to* consider when you choose an information collection 
method (see below) 

Shouid you use ability tests? (see p. 2) How afrcmt ques- 
tionnaires? (see p. 2 ) A*e interviews feasible? Mp.U ) 
A quick guide to pros and cons of each- metitibr (p- 5) / 
News about evaulation activities (see p. ^PX^ ' > y 



How To Collect Evaluation InformafioFr 

You can collect evaluation information in many different ways. If you need informa- 
tion to fipd out whether an innovative elementary school reading center has 

' actually improved student, reading abilfty*Jo»-;exainple, you would probably ask 
this question: H6w much have students' reading skills improved? ' 



jOuld use any of the following 



To provide a credible answer^ "to** the question, 1 you 
cinf ormafcion collec-tion techniques: \* \ 
" ■ ' - , '/w^ ? 

■ Give parents a rating scale 'to assess^ their children's Veading performance. 
» Send questionnaires to teachers trv get their opinion about student reading 

• performance. * >; '""^v / 

■ Interview students to tefc^Kebr opinion about student reading performance. 
» Observe students as tfre.y^r'ead and rate their reading ability. 

■ Have students keep a disry^ tfeeir progress. • 

■ Give students a natibnally* normed achievement tes#-that assesses reading 
performance* f \ "*%y 

» Have students tar^a^te^icrier-developed test. 

« Review student record* for achievement test scores, report card grades, .and ' 
teachers ' comments J^-y 

"This list illustrat^s^^alternative techniques an evaluator might use to answer 

the question, about Improved reading skills. They are interviews, questionnaires, 
rating scalers Candard observations, record reviews,' and ..achievement tests. 

To dtowe- r tire best technique for answering a particular question, you 
should consider/ four factors. First,* the method should be agreeable to your 
client and colleagues. If you want to use questionnaires, but the district's 
staff prefers interviews, you must decide just how serious the consequences 
of imposing your own choice, might be. ^ 

(more)' 
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How To Collect Evaluation Information (Cont.) 

Second , the information collection techniques should be technically 
, sound and the data collected from them should be reliable, valid, and target- • 
ed to the evaluation questions. Third, the information collection techniques 
should provide the best data the evaluation budget can afford, which means 
that you will have to decide such things a§ whether to buy or develop your 
own measures, and whether to use more than one technique for each evaluation 
question. Fourth , you must be sure that the metHods you choose will allow 
4 enough time for gathering and analyzing the data. 



Should You Use Paper and Pencil Tests Of Ability? 

Paper and pencil tests are among the most commpnly used measurement techniques in 
education. 

Ach ieveme nt tests measure competency in a given siibject. They can be developed by 
th*5 program or evaluation staff or you can buy them from publishers. Achieve- 

uest3 can be used to measure a student's knowledge of basic English usage 
v-r a class's ability to solve quadratic equations^. . 

The advantages of achievement tests are that they can be administered 
to large groups at relatively Tow cost and that carefully developed and vali- 
dated* tests are available in many subject areas. 

One disadvantage is that achievement tests must be properly validated to 
provide accurate information, and this can be a costly procedure. Another is 
that having high Scores on a test of factual knowledge doesn't always mean 
that the student! can apply that knowledge. 

Aptitude tests are measures of potential. The most common measure of aptitude is 
an IQ test. Aptitude tests 'have the same advantages and disadvantages as 
achievement tests. 



What About Paper and Pencil Self-Report Measures? , 

Paper and pencil self-report .measures ask people to tell about their attitudes, 

belief 3, feelings, and perceptions. Questionnaires, rating and ranking scales, 
the Semantic Differential, the Q-sort, and diaries are among the techniques 
most frequently uged insgvaluating education prograins. 

Ques t ionnaires are self-administered survey forms that_consistj?f a set of questions 
Answers, to questionnaire items can require free responses (short answers) or 
they can be structyed into "forced" choices (multiple choice items). Ques- 
tionnaires are frequently used in large scale evaluations to obtain partici- 
pants' reactions and opinions. They are less expensive to construct than most 
measures, but -the Jcind- of information you can get from them is limited, and 
people don't always answer the questions ttuthfully. Don't forget that you 
will have to follow up on those who don't respond, and that's an expensive 
undertaking. 

Rating scales can be used for self assessment or'for ^appraisals of other people, 

groups, events, or products. Student attentiveness, for example, can be rated 
on a five-point scale from 1 for not very attentive to 5 for very attentive. 
Rating scales are particularly useful when you need to- reduce judgmental data 
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What About Papar and Pencil Self-Report Measure 8? (Cont.) 

Co a manageable form. They are relatively easy to complete and they produce 
objectified data. 

Unfortunately^ they are subject to many types of bias— some raters are * 
lenient, and- others are not, and sometimes raters let their personal feelings 
influence their ratings (a halo effect). Further, the amount of information 
you can obtain from rating scales is limited because the rating categories 
are never perfect. , 

Ranking scales involve putting a set of items into a hierarchy according to some 
value or preference. Asking a teacher to rank four textbooks from one to 
. four according to their reading difficulty is an example. Like rating scales, 
ranking scales are easy to complete and produce objectified data. But remem- 
ber that rank ordering a long list of items is no fun and it takes a lot of 
time. Ranking scales sometimes ask peopirf to make distinctions among things 
' where they can't really see any difference. 

2 ~mzlc Differential is used to measure attitudes by relying on the indirect mean- 
-^gs of words. x — . 

For example, students night be asked to rate their country using a series 
/ sc\&n point scales like the following: 



UNITED STATES 

PASSIVE* — . ACTIVE 

SMALL . LARGE 

DEMOCRATIC " -r— — - UNDEMOCRATIC 



RICH 



POOR 



The Semantic Differential is relatively easy to com{rle£a^ it produces objecti- 
fied data, and respondents usually find it harder to choose "socially acceptable 
- answers than when they use an ordinary rating scale. However, the Differential 
can be difficult to score. * 

The Q-sort requires .Individuals to place a series of items or statements into rating 
~ categories so that some minimum number of items is assigned to each category. 
For example, teachers could be asked to rate ten textbooks as "above average" 
or "below average" so that at least two texts are assigned to each category 
(the remaining texts can be rated either way). 

Q-sorts produce objectified data and they force respondents to, establish 
priorities among items that are being compared in an evaluation. But the Q-sort 
requires people' to make very difficult distinctions, the directions are often 
hard to follow, and the resulting data can require complex analysis methods. , 

Diary 'techniques ask people to keep daily or weekly accounts of specific behaviors, 
attitudes, thoughts, or events. The critical incident technique asics people 
to record only those things that are particularly imfcorta^t, unique.,, useful, 
or* reve la tory . F o r ex a mp le, you could as k studqii^a tu keep a- dally, dla*y ~ oiT 
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the amount of time spent in free reading, or -to record the names of any books 
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What About ' Paper and Pencil' Self-Report Measures? (Cont.) 

they liked or disliked. 

Diaries and critical incidents permit people to describe unique situa- 
tions in their own words, but people sometimes forget to maintain them and 
they are often difficult to score and interpret. 



Why Not Try Observations? , 

Another information collection technique frequently used in evaluations is an eye .( 
* witness account of individual behavior or program activities. .You could use 
observations, for example, to find out which visual aids teachers are using. „ 
The information collected by observers can be reported bV checklists, rating 
scales, field notes, and summary reports. J ^ 

SS X tfit* observations require careful planning so, that theflLnformation obtained is 
• arsnrot*. Observations can give information collectors first-hand information 
.•about a program, and they are often the only feasible and economical way to 
•?ather certain kinds of information. But it is costly to train observers, 
and several may be needed to get reliable results. Another drawback is that 
people who know they are being observed may not behave normally. 

Time samplfofi observations involve repeated observations of a given situation. For" 
example, observers may note how many students and which ones ask direct and 
indirect questions during. ten consecutive five-minute intervals. ' * 

• Time^ sampling allows first hand observations of a program, and the many 
observations make it possible to identify unusual events that you might other- 
wise .think were/routine occurrences. When all the observations are made one 
after the other, however, they are likely to depict only one particular situa- 
tion „ and not the program as <a whole. " 



Are Interviews Feasible? 

An interview is an information collection technique in which one. person talks to 
.another or to a group. Interviews can be completely unstructured and spon- 
taneous, or you can decide ahead of time the kinds of questions >to ask. If 
you use multiple ohoice questions, even the response categories are predeter- 
mined. 

Face-to-face interviews might be used, for example, to Had out.why. paw^cipants 

dropped put of a program, and might consist of three basic questions with a 

series of two or more in-depth questions for each basic question asked. The 

best thing about the face-to-face interview is that it permits you to probe * 

sensitive subjects like attitudes or values. But these> interviews are usually 

Blmf consuming expensive, and you will have to give interviewers special 
, training. 7 • . % 

Telephone interviews also permit in-depth probing of .sensitive issues, and are less ' 
costly than face-to-face interviews. They are still expensive when compared 
to questionnaires, however. . You should also remember 'that not everyone has 
a telephone, and some people are reluctant ttf reveal their feelings or give 
pexsonal information over the phone. J 49 r „ nr ^ 
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Are Performance Tests the /fnswer? 



Performance f sts require people to complete a taak^or make some thing, and then 
, you aaseasthe quality of tha performance or product. One example of a per- 
formance test is whan -you have someone type a letter and then you count the 
number of words typed correctly in a set amount of time. Another example "is 
when a group of experts observe a teacher instructing a class and then use 
a rating scale to indicate their appraiaal of the teacher's ability. "C 
The major advantage of performance testing is that it relies on tasks 
that are close to "real Jrorld" activities. It is often very time -consuming 
and expensive, however, because performance tests generally have to be adminr 
istered individually arid they sometimes require the use of special equipments 
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^uid Record Reviews Be Enough? 



.* J . ES^JSS. n «an that you .collect evaluation information by going through pro- 
♦ tot vsUted documents. In a program where older pupils tutor younger ones, 
** /vimplei you might review attendance records to see if either the tutors 
.he children they taught came to school more r egularly after the urogram 

began7 ~ ^ r — °- - ' 

Record reviews are •"unobtrusive" in the sense that they do not interfere" ffi 
vithvthe activities of the program being evaluated. They can also be relatively 
inexpensive Jecause no new data collection is required. One problem is that - 
program documents may be disorganized or unavailable. 
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INFORMATION 
* ALTERf 


1 COLLECTION 
IATIVES 


V 

ADVANTAGES 


DISADVANTAGES- 


Paper and Pencil 
Tests of Ability 

7 

r * 


Achievement Tests 
Aptitude Tests 

• * 


Can be administered 
to large groUpTlft"~ 
relatively low coats 

Many published, stan- 
. dardized tests are 
available 


Expensive to~develop* \ 
arid validate '"' 

High scores do not 
necessarily imply 
thtt the tested 
knowledge can 
be applied 


Paper and Pencil 

Self-fUport Mea- 
sures 


Questionnaire 


Can be administered 
to large groups at 
relatively loir costs 

' : • r 

1 crn 


1 

v Can be difficult to . 
obtain sensitive 
information 

« 

-Respondents may not 
always' be truthful 

Must follow-up -to 
obtain adequate 
numbers* of respon- 
dents 

(Corit.) j 
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INFORMATION COLLECTION 
ALTERNATIVES 



ADVANTAGES 



— — r — — 

Easy td complete 

Produces objecti- 
fied data f ■ 

Haduces judgmental 
data into a manage-' 
able form . 

V . 



DISADVANTAGES 



Paper and Pencil 
Self ^Repoxrt Mea- 
sures <cont«.) 1 
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Semantic Differ- 
n entials * t ^ 



Rating Scales 



Responses may be 
biased because some 
raters are lenient; 
and others are strict 

Amount of information 
obtainable is circum- 
scribed by the rat- 1 
ing categories 

Halo effect N - 



Ranking Scales 



Easy to complete 

Produces objecti- 
fied data 



Difficult to rank a 
long list of items * 

Distinctions are 
called for that are 
not perceived 



T 



Easy to complete 

Produces objecti- 
fied data 

More difficult tg 
give "socially 
acceptable" re- 
sponses 
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Difficult to score 



\ 



Q-Sorts 




roduces objecti- 
fied data 

Forces respondents 
to establish* prior- 
ities among items 



Distinctions are 
called for that are 
not perceived 

Directions canJae 
too elaborate. 

Can requirt complex 
data analysis methods 



Diaries and 

Critical 

Incidents 



Permits people to 
discribe unique 
situations in 
their own words 



People don't ipaintain 
*them 

Difficult to scot 
# an^ .interpret 

(CoyltO 
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INFORMATION COLLECTION 
ALTERNATIVES - 



ADVANTAGES 



DISADVANTAGES 



Observations 



Standard 
Observations 



Time Sampling 
Observations 



Can observe events 
at first hand * 



Can observe events 
at first hand ■* 

More opportunities 
to observe 



Observers can change 
the environment 

Inter- and intra- 
observer reliability 
can be difficult to 
obtain . 



Observers can change 
the environment 

Inter- and intra-, 
observer reliability- 
can be difficult to 
obtain 



Interviews 



Face-to-face 
Interviews 



Permits indeptl* 
probing 

* Sensitive issues 
can be discussed 



Costly, 

Ipxer- and intra- 
rater reliability 
>e difficult to obt 




sl^ph c 



Telephone 
Interviews • 



Permits indepth 
probing , 



Costly 



Sensitive issues 
can be discussed 



Some people may not 
have telephones 



Less costly .than, 
face-to-face in- 
terviews 



"More- dlff luul trto- 
probe or discusjs 
sensitive issues 



Performance' Tests 



Record Review 
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Close to real 
world situations 



.Costly , 

Generally must be 
administered indi- 



vidually 



V 



Can require special 
equipment or apparatus 



Unobtrusive 



n be inexpensive 



No new data collec- 
tion required 



.Documents may be 
disorganized or un- 
available 
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News Briefs 



The essentials of an adequate evaluation are spelled ou t In no uncertain terms by 
G. Kasten Tallmadge of BMC Research Corporation in a new "Ideabook" fd*. 
the National Institute of Education. Did a change occur? Was 'the effect 
' consistent enough to be statistically signigicant? Was the effect educa- 
tionally significant? Will it work just as well in other- places? Are you t 
sure it was the program that made the difference? Is the evidence under- 
standable and believable?- To top it off, you'll find. a lucid chapter on 
common evaluation hazards and how to avoid them. 

A limited number of copies of Ideabook are available from the National Institute 
of Education. Education and Work Group. Washington .' D.C.. 30208. 

• V/ . \ 

Nearl y 200 programs that measured up to Tallmad ee's stringent criteria are described 

i p a 200-page paperback called- "Educational Programs That Work. Exemplary 

projects chat have been approve* by the Office of Education/NationaJ. Institute 
of Education review panel include early childhood education, reading, .math, 
•*72c.i.J. ed, career ed, environmental ed, alternative schools, and more. 

•j.dex by sending prepayment of $4.95 per copy to Far West Laboratory for Educational 
Research and Development. 1855 Folsom St.. San F rancisco. CA 94103, attention 
, Order Department ♦■ x "'.*'*" 



Th e Evaluation Network offers up-to-the-minute information on Resources, training 
prograns and materials , new developments in evaluation technology and theory, 

and practical applications. Membership includes subscription to Evaluation 

News , a quarterly publication edited by Michael Scriven. 

For further information write EN c/o Thi Delta Kappa, Eighth St. and Union Ave., 
, PQ Box 789. Bloomington. Indiana 47401> A nnual dues: $5.50. 



The Evaluation Improvement Program i s a nontechnical insgrvice_e ffo rt to train teachers, 
• ' StSSmL and agency *" **« t^hniaues of education • programming 
tion. Tit. off era self-ins tructfonal material^ including a Program Evaluator s 
Ouide, an evaluator ? s workbook and a trainer's guide, and „a catalog of evai.ua- 
tion training materials from the National Institute of Education. . - . 

Contact W.W. /Walton, Evaluation Improvement Pr ogram. PQ Box 2845, Educational 
■ Testing Service* Princeton, NJ 08540 . • 



Dr. Arlene Fink and Dr. r Jacoueline Kosecoff . editors of this newsletter, will con- 
" 7 d uct two workshops this spring on How To Eva luate Education Programs, .The 
- dates are April-24-25 in Washington and May 11-12 in San Francisco. Mark your 
calendar now and write or call John Ekberg, ADD. Capitol Publications , Inc . ,, 
d«^.'«i««4 9 Ave\. NW. Washington. D.C. 200 37. Phone (202) 45Z-1Q0U. 



Evaluating an AGE Program 



\ 



Wayne D. Dvorak and Donald P. Lang 

Any evaluation plan designed to evaluatef\ Evaluation are the descriptioi 
arts in general education programs, if it is to 
be effective, must reflect a reconsideration of 



the whole process, , of education evaluation. 
AGE programs represent a critical philosophi- 
cal shift regarding the place of the arts in gen- 
eral education and the values and special .con- 
tributions the arts can make to a school pro- 
gram. The approach is significant because it is 
global — the arts are used as oife means x to 
teach everything. The evaluation plan, there- 
fore, must be designed Co reflect that univer- 
sality—the philosophy, the aims, the out- 
comes, and the complexity of the program it 
purports to evaluate. It must be compre- 
hensible and flexible, yet retain validity, 

Education programs, particularly those that 
^are characterized by hew approaches, will ul- 
timately be evaluated by everyone — project di- 
rectors, participants, teachers, students, par- 
ents, and administrators. What is not needed 
is an evaluation plan that reduces complex 
and unique education programs to esoteric jar- 
gon and statistical "simplifiers" and descrip- 
tors Uke "x," u y" population scores, and stan- 
dard deviations. 

The evaluation plan proposed here results ' 
in an understanding of the program and its ef- 
fect upon the participants. It can be and has 
been* used effectively to evaluate either arts 
education or music education programs. De- 
veloped by Robert E. Stake, director of the 
Center for Instructional Research and Curricu- 
lum Evaluation at the University of Illinois/. 
Urbana, the plan is called "Responsive Evalu- 
ation." It establishes some important alterna- 
tives tp the whole process of education evalua- 
tion research, particularly of arts in general 
education programs. 

Raaptmshre evaluation 

The two major activities of a Responsive 

Dvorak is assistant prof stop of music education at Hartt 
Colli* of Music, University of Hartford, West Hartford, 
Connecticut Lang is assistant professor of music educa- 
tion, Unhrersity of Delaware, Newark. 



and the judg- 
ment of the program under examination. The 
difference between "description" of the pro- 
gram and the use of "descriptors" is an impor- 
tant distinction. Descriptors are simplifiers. 
Descriptors only tell, for example, that certain 
"individuals are observed, found to differ, and 
the distribution of the stores is d&qibed* 
Covariations of various kinds are reported and- 
interpreted." 1 Description portrays." Descrip- 
tion tells us" what the program is really like. 
Responsive Evaluation portrays the program 
in all its scope and complexity, It is based 
upon ". . . what people do naturally to eval- 
uate things. They observe and react" 2 The ap- 
proach is not new. What is new is the begin- 
ning of a technology developed^ around this 
natural -behavior, in part to overcome its de- 
fects. The accompanying figure is a jpaphic 
representation of that technology. 

Description . " v 

Initially, statements' concerning the in- 
tended antecedents (from where the program 
is starting), intended transactions (what class- 
room procedures will be used), and the in-* 
tended outcomes (who will benefit and how 
they will benefit) ore gathered by the evaluator 

, from program personnel. Next, he or she ar- 
ranges for observations of the program by Vari- 
ous people, including himself and individuals 
with .no vested interest in the program. He 
gathers their perceptions and impressions of 
the 'observed antecedents, observed proce- 
dures, and observed outcomes, remembering 
to take into consideration the differing value 
perspectives of each observer. \ 

The diversity an ACE program demands 
redundancy in the evaluation process. Repeat* 
ed encounters by the same observer, by many 
different observers of the same experience.^ 4 
and a widening perception by all can "discov- 

. er" significant happenings, either favorable or 



tRobirt E. Staka, "Raaponaivt Evaluation" (Urban*, Illinois . 
Cm* far Instructional Rwt tfc h and Curriculum Evaluation. 
W2),p.4. m \ 
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unfavorable, that may be missed in a single en- 
counter. What is of f value within the program 
and what is not Will emerge as a consensus 
among the observers. 

At this point it is important to note the pos- 
sibility of using appropriate tests as part of the 
data-gathering process for establishing ob- 
served outcomes. The choice of these in- 
struments is made as a result of observing the 7 
program in action and of interacting with vari- 
ous groups that have an interest in the pro- 
gram. 

Judgment 

Education programs are rarely simple- 
most assuredly, arts programs are complex. 
Eacfc observer and participant of the prd§ram 
brings to his record of the program his own 
biological inheritance, his. past learnings, and 
his perception of the immediate situation./ 
This becomes the "truth" of that situation fdr 
the observer. Sines there exists no single, ulti- 
mate truth regarding the value or relevance of 
any specific education rdgram (preordinate 

aluators notwithstanding), the gathering of 
many different versions of the truth according 
to differing value-perspectives is faore ef- 
fective, revealing, and reliable than the pur- 
suit of a single, ultimate value. Divergent and 
even contradictory subjective evaluative com- 
mentaries can often generate valuable infor- 
mation about how an AGE program really 



works, as well as contribute directly and sig- 
' nificantly to the evaluative phases of the re- 
search project. 

If the description is accurate, and complete 
if the reader of the final report can get into the 
heart of the program, if the program personnel 
can be shown "the realities of the program, 
rather than what they believe the program to 
be, then a basis for critical judgment and 
meaningful evaluation of the program has 
been provided that goes beyond the inter- 
pretation of test scores. 

The total number of value perspectives gath- 
ered and reported is limited only by the re- 
sources, the time, the priorities, and the inter- 
ests of the evaluator and his clients. Proper at- 
tention to diversity will allow for the greatest 
/ service to the greatest number of people. It can 
also help to counteract the reports of the ob- 
servers or participants with "an axe to grind." 
an important consideration in any style of re- 
search reporting. 

In the judgmental phase of ttife evaluation, 
the evaltiator notes the discrepancies and con- 
gruencies between the intended antecedents, 
procedures, and outcomes. Either or both of 
the intended and actual antecedents, proce- 
dures, and outcomes may also be compared to 
an ideal or accepted standard. Similarly, the 
evaluator may note if the intended antece- 
dents and procedures could be expected to 
produce the desired outcomes — logical con-' 
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tingency. Alsq, th'e evaluatbr may note the 
cause-and-effe£t relationships among observed 
v antecedents, procedures, and outcomes. 

Any complex, diverse program that haa_hadL_ 
repeated viewing will yield ambiguous and 
contradictory information. Traditionally, this 
has been the bane of preordinate evaluators — 
the results do not fit neatly into the paradigm. 
The Responsive Evaluation approach over- 
comes this dilemma through the use of what 
Stake calls "adversarial procedures/' 

Most common in the work of legal theorists, 
adversarial procedures call for the reporting of 
evidence about a given program by two indi? 
viduals — an advocate and an adversary. The 
advocate's statement is a summary of the most 
positive claims that might reasonably be made * 
about the program, while the adversary sum- 
marizes the most damaging claims. Neither 
statement is intended to indicate the personal 
'opinion of its author. Each statement presents, 
rather, the claims that might be made about 
tLa strengths and weaknesses of the, program 
V ad upon the evidence gathered during the 
. a'^'^uve process. In this way, the , 
iA ...d value of the program emerge to 
4 ^3e peopie who must assess its worth and 
must judge it. 

Responsive Evaluation has yet another im- 
portant tfcataefit for the evaluator and his client 
Because^it is not restricted to elaborate para- 
digms and a statistical format, the final report, 
can be written in a natural communicative 
style. There are many examples of the natural 
style of evaluative reporting that can stimulate 
the reader's interest in evaluation as well as 
educate the reader in alternative evaluative 
process. They include reports by Stake, 3 , 
Brauner,* Lang, 5 and Dvorak.* Each Qf these, 
exampi^JHhphasizes the service function 
\Nr^^r<riian the research function of evaluative t 
repottmg. An evaluation must provide useful 
. information to a client The results may* or . 
may not be generalizabie to other situations, 
but they must be valid for the individual pro- 
gram under evaluation. * * 

* 

Summary 

Even though the evaluation plans tfndflnal 
reports of the Responsive Evaluation model 
will vary in direct relationship to' the diversity * 
of programs being evaluated, the following 

iRobsrt E. Staka and Craig CJtrd# t "An Evaluation of TCipr. * 
tht Twin Oty Instttutt for Taiantad Youth. %97l"AERA Mono* 
papti Smim on Curriculum fiVoiuotion. Vol 7 (Chicago: Rand 
McNaUy. 1974). 

*Omim Braunar. "Tha First Probt," AERA Monograph Sariat 

- on Curriculum Evaluation. Vol 7 (Chicago: Rind McNally. 1974). 

Donald P. Lang. "An Invtstigation oltha Egactivtnata of tha 
Mathodoiogiat of Music Instruction Uaad in Saiactad* Schools of 
tha Watt Hartford. Connartimt School Syttam." unpuoliaaad 
JuLtofridliaangrinn(WairHlfhrrf rnrmtrtimf Htm 
Music Univarsity ot Hartford, 1978). 

•Wtyot p. Dvorak. "Or* Music Education for Disndvantsjad 
Childrafr-Unguaga Arts and Mathamatlcs Through Music** ml* 
uatioB rtport (Hartford. Counacticut: Coanacticat Statt Dapart- 
ia«»to< Education, 197S). 

» 
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principles are common to all: 

• Parity. The value of an education program 
is determined hy people, not the research 
design. 



Ubiquity. Evaluation is a partner to all ed- 
ucational endeavdrs. 

• Diversity. There is no single, ultimate 

• truth in the value of any education pro* 
gram. 

• Utility. Evaluation must be a service to its 
clients. 

• Redundancy. Understanding of any edu- 
cation program comes with repeated en- 
counters. * f 

• Ambiguity. It is nether necessary nor de- 
sirable to force a consensus about the value 
of any education program. 

• Generaiizability. Important in the work of 
preordinate evaluators, generaiizability 
may not be worth the effort Each program 
must be evaluated on the basis of its unique 
aspects. 7 t 

In his bodk, Zen and the Art 0/ Motorcycle 
Maintenance, Robert Pirsig reveals a new and 
thought-provoking way of looking at some of 
the premises upon which Western thought is 
based, including those thought processes nor- 
mally associated ygith education, research, , 
and especially th&scientific method. Stake fre- 
" quently has said that Z?n and the Art 0/ Mo- 
torcycle Maintenance is the best bookjOn edu- 
cation evaluation that is yet available. The ap- 
plication of some of these ideas to arts 
education 1 evaluation is not only relevant, it is * 
essential. As. Pirsig reminded us: 

It's been necessary since before the time of Socrates 
to reject the passions, the emotions, in order to free 
the rational mind for an understanding/ of nature's 
order which was as yet unknown. Now it's time to 

' further an Understanding Qf nature's order by reas- 
similating those passions which were originally 
fled from. The passions, the emotions, the affective 

^ domain of man's consciousness are a part ofna- 
ture's order too. The central part 1 

.Maybe these ideas only work in literary 
form (fiction at that) and are not applicable to 
evaluation of AGE programs* But as Stake 
himself has said, . . were he to have greater 
ties with the anthropologist, the journalist, 
arjd the poet, the contemporary evaluator 
might have dealt himself a more responsive 
assignment/' 9 The. analogy is clear. For ef- 
fective evaluation of AGE programs, the logic 
is inescapable. ^ 



ftobtft &_SUk*. "Tht Savan Principal Cardinals of Education- 
si Evaluation." handout for praaantatioo at AERA annual matting. 
Chicago, April 1972. 

•Robatt M. Pirsig. Zan and tht Art of Motofcyclt Mainttnarica 
(Ntw York: Bantam Books. Inc. 1*74), p. 287. 

♦Robart E. Stakt. "An Approach to tht Evaluation for Instruc- 
tional Programs (Program Portrayal vs. Analysis)." paptr otliv- 
artd at AERA annual matting. April 4. 1972. p. 1. £ g 
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TCITY-1971 EVALUATION REPORT: CQilTEITS 



T?I>e^e1^ng and People 
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A PartLtrfpant Observer 1 * Persp^V ve 
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EVALUATION PLAN ? ' ' ' . ' * 

The evaluation plan for «%3^![•S,^5^^1,^ 0 b,**! , "^•• 
irtlcle, "The Countenance of ^uoaWona 1 tv ai J ment af numerous 

« calls for (1) a th ° r ° u S\^i?e The plan emphasizes the col- 
personal l<*vr&'& backgrounu against wnich the Institute acti- 
vtflesVaKeTlac™ tHe acUvUies twelves, an d the results. . 

, Be cau,e the Institute is of ^^{^i^r^r 

US areT„sensKlve e tf -^^^H^^S SET 
here wer% 'gathered by direct observation, ^chi Jmpor tant 

no? used. . *J a matter of c °^ e *J.Yw1US^ S than for the indi- 
Sfget a .picture of the results for a who^ 01 ^ ^ teacher and 

vidua! members, of tins ciass. °V U 

evalSator views were collected- - . ft ^ ^ 

The daily activities of the ^/ h fn^ 
correlated with the activities of ge J orm | tive -^valuation-.* 
^n/?^ ^denc, and solve pro- 

eedural problems. 

' For the "outsider" this cooperation raises the ^f^M^T 
iectivitv of the final report, To P JC&DDrt and an ad*er- 

■ .rr^roneTuL^ 
these conflicting arguments. • 
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tcity-1971 evaluation report: THE SETTING AND PEOPLE 



• * — / 

The i971 session of the TWIN CITY INSTITUTE FOR TALENTED YOUTH 
was held on the campus of Macalester College in St. Paul, Minnesota. 
Numerous activities extended to the urban areas nearby, out into the 
state, and even to West-Berlin and Mexico City. Students back-packed 
across Isle Royal, dug for Indian artifacts ,at Port Sweeney, visited 
Twin City and Chicago industries, and talked to officials at City 
Hall. Still, most Institute activities took place in typical class- 
rooms, at Macalester. See map below. 

This was, the fifth summer. Previous Institutes had been -held at 
Murray High in Saint Paul and Marshall University High in Minneapolis, ■ 
and at Macalester, and Augsburg College in Minneapolis. 

• tf • • ' > 

The Institute is a summer school experience for specially talented 
youngsters in the ninth through twelfth gradps of Minneapolis and St. 
Paul schools. It is supported by these two districts, by a substantial 
.^rant from the Harington Foundation*, _and by numerous donations. No 
cuition is charged. 

Al'out *?0 students attended in 1971. The number h.as been *rowin<* 
s^s .--Nr. The length of the terra has increased too,„now up anouier 
v \.z seven weeks. The 1971 term extended from June 15 to July 30. 



The teaching staff consisted of 28 master 
teachers and 28 associate teachers. Master 
teacners were selected from schools through- 
out the metropolitan area and beyond. *The 
associates were selected from the two urban 
districts j< Additional teaching help became 
available from University of Minnesota stu- 
dents seeking experience, stimulation or 
course projects. Thre* administrators » two 
e v -iuktors , and a secretary alsojtere sta'ff 
members. < " ' 





Class Spaces 

35 -Amerioac Studies 

38 Archeology # N 

39 "Astronomy 
39 -Chemical Be 
38 . Computers 
3D Computers 

ScieT 

36 Dance 

38 Ecological 

. v Biology 
35 Environmental 

Accounting 

25 -French 
25 German 
2 Graphic Arts 

39 Mathematlci/for 
the Disenchanted 

55 ' Magic 

35 Poetry ' * * 
"|lf Russian - 

2 Sculpture 
20 Sesame Street 

H Spanish 
35 Theatre 
35 The Urban Hero 

2 Wilderness 

Lc <dership 
3S Writers Workshop 

31 TCITY Office 

3 - Library, AV Ctr 
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TCITY-1971 EVALUATION REPORT: G0ALS 



The primary objective of t he Twin City Institute X;; „ttf errato an 
educational, program that half^'lI'Ultg^oadeinio ami social appeal for / 
students who possess a variety of artfSJtic, Janpua^o, :wtiMill f K* ana 
leadership talfents. The program U? not aligned to repeat tho regu- 
lar-school experience, nor to repair it for the disenchanted. The 
Institute is 'designed to create a 'special experience, cutting across 
tudent interests, group identification, idea exploration, and the 
aditipnal school curriculum, > 

4he Institute' is willing to takJ risks. Teachers will develop 
courses that have their locus injtneory or intuition rather than in 
more conventional curricular constructs. The staff will recognize 
that any program, traditional or experimental, that chooses to work 
in an atmosphere of- freedom, where trust is extended in social re-, 
lationships, and where^new ideas are encouraged, is going to be 
vulnerable to charges of aimlessnes^and confusion. 

The Institute s v taff — particularly teachers involved in such areas » 
as the arts and social scienceb, where truths are more>< sub jective 
and where curricular goals are disputed — will develop programs that 
emphasize inquiry. To' create a compelling and stimulating learning 
environment, TCI teachers will develop problem-centered courses 
which encourage thought, inquiry, and creativity. 

Finally, teachers will introduce, students to a total Institute 
environment. The Institute campus will be open to the young, sen- 
sitive, inquiring mind. Students will "Share with each other — 
through conversation and exhibit-'-their productions and products: 
telescopes, dances, poetry, music, foreign language fet^s, newspapers, 
etc. Against' a background of computer programming, art, pottery- 
making^ dance, athletics, folk danfeing, games, canoe building, stu- 
dents will meet and talk and create. 

\ 

Having come interested in math, science, poetry, or archaeology, 
students will leave more committed to understanding and appreciating 
the total Qoncerns of men. Success for the Institute will mean that 
students and teachers have infected each other with a personal hon- 
esty and will have demonstrated that learning can truly be humanizing. 

Charles Caruson, Director, TCITY ^ 

Goal Eyaluation . EvilUators have an obligation to raise the 
question, "were ,the right goals" pursued?" Different people have 
different ideas, of course, as to what the right goals ar£. Still, 

goals and priorities should be evaluated. 

» 

Many teachers and curriculum specialist's endors^ TCITY's .in- 
creased emphasis on humanization, personal awareness, and problem 
solving, and decreased , emphasis* on skills and knowledge. Some 
teachers and many parents disagree, wanting the school experience 
to pay off in answers to«.the classical academic questions — the /kind 
that* get students "employed, admitted to college, etc. Tn fch* eves 
of tty* eval^atnrs^ thq. TrjTY j^te^S- * r e worthy goals > suitably Vt is- 
cua*ea ana reasonablv operationajLlzed. 

• . * 

* . * 160 Robert Stake, Evaluation Specialist, CIRCE. 



TCITY-1971 EVALUATION REPORT: 



/admissions 



in £ fl residing in Minneapolis or St; Paul during 1970-71 and 

S??v itlb W6re eli « ible ^ the 1971 TC'ITY Sir Insiuiti 
City scSolf?o J Asso e iate Director, Robert Rose JisUed Tv,lk 

placed orrec^u?t?n ff S?^ aS \ StU 5 entS to appl y- Some emphasis was 

veroai and numerical scores from the Differential Aptitude Test 

Some courses, like Wilderness Leadership and SesSme Street had 
far too many applicants for the number of Openings. A few? like • 

'or-third e nhoi ,nder " SUbSCribed - Student3 ll3?in « music as a'se^nd 
or third choice were added to rosters. 

hal J r ; S arus -° n and Mr - Rose examined the f ormsViNeking especially' 
If student3 wlt h DAT scores over 75 and ajS considering low 
achieving youngsters with no marked behavioral problems whosf tea- 
counselors felt that the Institute woSld draw'ou£ Ihell 

loll of thi e ?o fl S, H Ab ° Ut 2 ?° S f th6Se " Special Admits " *ere acceded. 
bu? the ^snon^MT^ 6 ^^^ a PP lication 3 and made recommendations, 
SarusSn S^""* bUWto " ° f selection ^inly upon 

tha? 5 I™ t he ,l 21 * 7 aPPii^ants were accepted, with the expectation ' 
2S£j?T W ° Ul ? ^ cel - 0f thi3 number,. 771 quickly accepted. Some 
?o" ! nt n a ° c fP- te d later, and a few students who had' been in the »al? 
( ternate" status were added to the accepted list to mi^oS classes 

K It was expected that 775-800 would attend. classes. 

to sL^?*-'/""?. 1 !!' 838 s - tud ents appeared at Macalester College 

IZrZtS n l h L lnSti ^ e ' Durlng the couf,se of the 7 week's about 7 
percent of these students disappeared from their classes, leaving 1 
approximately 800 on the final days. These were pretty evenly di- 
vided between Minneapolis and St. Paul and split S 5 0 to 350 In fa- 
■ of TC^TY lll'l ,?Sf k5 3Cholar3h iP students. About 16 percent * 

* attended'^ 19I9 3 W6re repeatera from ^Oj 1 percent had also 

' , 7« a^H? 6 ? 131 ?" fc f a ? mlt eighth « rade rs this year' led to problems— 
7J0 additional schools to contact,- new counselors to involve, and an 

ihlTtl Hf<, DAT T SCOreS .- s ° me , teachers thought 8th graders were less - 
a omt,n?. d J S P 3^t th f m ? elVes » and a few students complained about' 
SSKSS at i ond J f f ^ulties, but the consensus was that the younger 
6 f a ; p f ed well and caused few problems. Some largerciafses 
duping tha£ age?" experience was a. better criterion for sub^ 

Any program for "talented", youngsters is potentially faced with 
< Charges of elitism and racism. Many teachers thought there was less 
t elitism and snobbery at TCITY than in honors classes in regular ' 
school Minority enrollment was, perhaps, 8 percent. 

Admissions questions that cpntinufe to be discussed by the directors 
and the TCITY Board: Is there a better way to identify talented stu- 
aents, than by test taking and faculty recommendations? How large a 
_ percentage. of "Special Admits" can be enrolled without endangering the 

• academic spirit of the Institute? Are the admission procedures con- 
^ sispent with the philosophical goals of the" Ins Utute?' 

y& • • i6i • 13 



TCITY-1971 EVALUATION REPORT: CALENDAR 



A calendar \>f ow.it ;t tVr '.VITY- |o; i 
Serin* '^f^.^anni,,,' t>y ,„ a , t; V „,,,,„,,.,; 

Way v 15 Macalester College cbq^Q ni^*- V ■ , . \ 

July 1 Dance and Muslc^lSbined ^o present a^oarm- f r"" 1 
classes crowned a ^^fp"^ a progrc-am; Russian 

July 3 0 InltuSrindld? 1 ? 8 for paints! ^ 

y institute. ended; Language groups performed fqlk dances ' 

• , -Monday, July 1 9 : A typical- day at the Institute 

BialogJ classes wenfto St pL?°££ e - 0n H an .? « ay back-packing trip, 
physical tests? . P&Ul Ramsey H °spital for a series of 

■m^S 8:15 or ' 8: 30 other classes smarted in classrooms on the 

^|o le anr t n^cont,„ H u°K SntlT^^or^T *£s££ "5** 
started, on a bike trip along the i.iss?ssI P p Tilver at U-fS*""!^ 

frl.bee. foftbal^.anS^t^^St'Hn^^ t6nnl - S ' -. 

^l^ t £4c t e Ud o?L% b ^ n ir?ne tr o U 5a 1 ^? 4e ° a " 10 •*"»• > 2 ^ 
«rfUTn g W ?en 3 e\ U « 

«ere finishing their redwood and fiberglass %ano^f ?o«™ . 
were In the office duplicating their "Iroaaslfes" f or MsSlSuS™ 
to people in the streets, and there, were other activities? • * 

n.S y „i :3 ° mo3t °£ tne 150 students who stayed for afternooli activi 

6 16 2 • 
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tcity-1971 evaluation report: CLASS AIMS AND ACCOMPLISHMENTS 



2^AN STUDIES ' ^30 ' students 

waster :Qene Lohman; Assoc :Bob Niemela 

Aims: To examine "American" ex D eri- 

IS5 e *! !! evealed "through histories- 

SS/ P f? rmS * To relate contempo- 4 
rary problems to,American traditions 
and r more. * 



SfffSfJ, Built a group ' iet ifc Plan 
activities.- used thematic rather ' 

iSStS hr ?S?J 08l S al Study of ■Awrrioan 
• MaHn *• lowed such themes as "the 
Making of a President", "packing 

5£L? Upreme . Court -'' "important 

trips?" SlnCe 19 " 5n ' To - ok 

Contoent: Sustained high level of 
interest and excitement, students 
involved' more in interpretation— 
often with too few facts— than in 
acquisition of facts. Teachers," 
etudents were, open, honest about 
tclnl ignorance and bi^-a good. 



'CHEMICAL BONDING 1 38 students 

Master: John Edwards; Assoc :Don Land 

Aims: To establish electron pat^ 
terns of atoms. • To relate, these 
patterns to the periodic table. To 
relate the ^lectron distribution 
pattern to spacial arrangement* that' 
can be demonstrated by 3-p< figures.' 
To use determined pattern >of elec-* 
tron distribution to predict chemi- 
cal formulas. To demonstrate the - 
shortcomings of having a single 
model. ; , 0 

Sketch: .Top tea covered and under- 
standings; reached.. J.ab used to de- 
monstrate tlu^ actual making,, of. 
products-, disputed. 

Comment: Took younger students, 
^. showed that 1.4 year olds— though' 
. sometime left breathless— can learn, 
complex chemical, concepts. 

V " '.."•* - 
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FRENCH " ' . ' 36 stud \ 

. MasterrBarb Gunderson ; Assoc :D.- HoSen 

Aims: To learn French through*, simu- 
'ini T °f experiewfeV^n. France, to . 
: compare life in France and -.the USA < 

CanadS? r ' the uni< * ueness of French- 

2f?Jj h i \ field test ^r simulation 
Wits developed by U of M people, - 
^•g.', preparing French bread, pre- 
paring crepes, playing Bridge, 
- Cyrano de Bergerac-. Weather Expres- 
sions. ... • _ 

Comment: Teacher preparation excel- 
lent, a fun class. 

SCULPTURE/ POTTERY , 37 students ■ 
Mas ter : J . Fontaine ; Ass oc : P . Fi t zger aid - 

Aims: Tq create an .awareness of the 
relationships between man, his en- ' 
- vironment and his art; a greater 
perception of art and the environ- 
ment; a greater sensitivity to form 
, color, and design; an introspection' 
into why men make art— and why each 
student does. ./ . 

Sketch: Students at work, creating 
tnre^lmensionaE'.art objects of 
..'.clay, plaster, pap^r mache, metal, / 
.styrofoam; potting,welding, Jewelry 
. makln «- ' ». -r, . 

Cpmment: For most .students a new 7 '' 
experience, a personal expression 

< 

1 COMPUTERS in SCIENCE ; 28- students • 
Master : JohnCrockerjAssoc :.PaulGif£$>rd 

Alms: To solve science problems 
using the computer. Emphasis on 
. simulation. Examine career opn6r- 
tunities. f ¥ P 

- Gommeitt: The highly successful 
mini-course, canoe. building, also 
.taught by Crocker /and Gifford ,'al- 
f -.most swamped, thiy course.' 
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ARCHEOLOGY . » 4 students 

•Masters: Dan Conrad, Bill Hobson 
•Assoc: Skip Olson, Dan Wills tte, 
Jan Peterson 

* 

Aims: To participate in a real 0 
i<4 science field study, relating 
the methods and. concepts ahd issues 
Of the several sciences. -To exca- 
vate a village site near Port 
.Sweeney to professional specif ica-' 
tions. To consider the life and 

• culture ofvthe Mississippian 
•« Indian. 

Sketchy At the instigation last 
.spring of the master teachers, Con- 
Zfi^Si Hobson, r Mr.- Carusdn arranged " 
with the State, Historical Society to 
excavate portions of an Indian site 
near Pbrt Sweeney. The Chief 'state 

• Archeologlst was reluctant to risk a 
d£g by high school students, but his 
colleague, pavid Nyatuen, saw its 
edupational fieri t and got the okay. ~ 

v Les Peterson, a younger .jj taf f arche- 
oaogist, accepted responsibility for 
^supervising the dig. 

- 

N Members of the TCITY staff and a 
, Party of students spent a May week- 
\ end. at the Port Sweertey site for 
orientation and training. They dis- 
cussed' the. historical, social, re- 
ligious, and< ecological aspects of v 
Mississippian Indian life. They 
considered the technical and. ethical 
aspects of a 5 week encampment at 
the. Welch Village ski chalet.. Plan- . 
' nlng at that point, was careful and 
detailed. 

During the first week of the" Insti- 
tute fthe "Special Math"" class— using 

.precise instruments— surveyed the 

•Site. v A village was expected there, 
near the visible burial mounds. The 
village was the target, not the 

.mounds; About a dozen ■ 10x10 foot 
squares were randomly selected foV . 

, excavation. 



At the .beginning of the second week 
^ <W arch'eologlcal. students, 5 TCITY* 
staff members,, .several Education 
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, students ,from t!ie University of Mih- 
nesota, and Let bopah vl4g,:tn,-. . Tho? 
au t : witn Ktvwoht, caroru+lv ivivm-' 
i m»- the a I t*t , .•xanMnlng tiu- 
faee;t. Slowly, ivi'U p.i I::fu I | v :|i,*.<- 
l lr:«l Week.:, Ihi< :-.tu\t«Mtt(t U*weii\| 

ni«; :uuT:u'i iiu-ium. Nothing, 
artitacta. None of tho e.xpee 0 ted 
tell-tale postholo dlsntoloraUotw;. 

n^ 7 ^ 1 ^ 6 ' The " fact that had 
provided, a basis for revisionof the- 

archeological map, only partly abated, 

their- disappointment. 

During the evenings they, played vol- 
leyball, canoed, swam, partied, dis- 
cussed digging techniques, revised 
camp rules, (broken first by teach- 
ers longing for a Beer), made clay 
pots, took pictures, ate; talked ' 
about the similarities- and differ- 
ences between grave-robbery and 
archeology,.-. .One evening they in- 
vited, families from neighbor ine 
Welch Village to share a pot luck 
supper and songfe,s.t. fh% evenings', 
went, well. ' 6 



In the fourth week, digging adjacent 
to the.mojjnds, they unearthed pot 
chards, missile points an\i other 
artifacts— not of the expected Mis- 
sissipplans but of "an earlier In-, 
diarv, the Woodland Indian. The 
find, sjowly realized, was a basis 
for revising both the local archeo- 
logical map and calendar. The 
archeologists wer.e as plea§ed as the ' 
students. 

In the final week on-site, following 
Les Peterson's evaluations, the* dig- 
gers moved closer in. The plan was' : 
to avoid the middles, of mounds, but ■> 
to dig at their edges so as to de- 
termine the 'age and circumstance of 
the burials. On the evening of the , „ 
next to last day on site, the dig- 
gers were confronted by a , large 
group of Indians, who u identif led 
themselves as members &f the Ameri- 
can Indian Movement. Outraged at 
seeing Students near' the- graves 
they shoveled dirt back into : the - 
excavations., even as the startled * 
students^ scrambled out.) Some- stu- 
dents .weae frightened /ff ; some 
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pleaded j*or -talking it over. The . 
AIM members went on to burn field 
•notes/ pictures, \ tools. OneVTCI 
student Was physically . shakeo, all 
were*, emotipnal ly ^shaken . 

Some students and their antagonists 
returned to the 'Chalet. More film 
and diariep wer*e. taken, but ap- ? 
parently nothing else.. A student 
anked the AIP^ people to- :;tay for - 
dinner.. # They -accepted ; then the 
i;ook announced she cou id ,not. feed 
them all. * Further talfc <TTd nbs^ood . 
The AIM demanded ahandoment of the 
:;ite,by 9 p.m. Tho camp was cleaned 
up and vacated by 8; 



JR6AN HERO • * *k students 

Master: S.Sandell; Assoc: R.Pest611o 

Aims: ' To search for today's Urban -, 
heroes, to contrast ^them to the John 
Waynes, Charles Lindbergs and George^* 
Washingtons; to - j£j s s^it the city, to 



learn of urb,an survival, of power^ 
of men. * 



Vpe Twin Cities press candied the 
'> tory on page one. The Indians 
v r>ed, "sacrilege!" Mr. Caruson 
* nsvsred; !t Ypu'hejed to .understand a 
. '-Jure in ohder to honor it." S6v- 
: . ci<xj 'i later some of the Indians 
£ j campus to discuss the en- 
counter with the students. To a 
, Large degree they were able to reach 
a reconciliation. ' * 



Dketch: , Ctudents first developed* a 
cla^c-community ; they became aware 
of the personal, social, and econQ- 
mic motivations of the city; they 
developed skills to express their 
idoas in a creative mediuta (g-.g.* 
photographic essay, guerilla the- 
* atre); they explored themes (el^., 
I \ the individual acting Recording to 
J \iis convictions in a man-made en- 
vironment, value conflict and con- 
flict resolution). 

/ Comment: Teachers imaginative, 

sensitive; class took an incredible 
number of valuable field trips. / 



Comirfelit: For ' these students this 
was % trying, -exhausting, fulfilling 



summer. % The issues and struggles . \ Airrfs^ Painting, drawing, explorir 
were serious— they .learned a great \ n IiT m edia. "We will approach art e 
at archeology ^ a bout commu- ^-Considering each student 1 ? personal 
nd about themselvesTV talents and interests." ' CreatiM. 



deal abo 
nlties an 



GRAPHIC ART , 36 students 

MastertGopal Mitra;Assoc:Robt .Horton^ 

loring 
by 



RUSSIAN . 35 students 

Masters: Pon Ryber*g; Cath.Filipovich 

Aims: x To, experience/ t^o pondfir the 
Russia^ way of life — communication^ * 
incentives, political system, cul- 
ture. To leafrn some/of the language 
through informal discussions, -jfole- 
playing, clas-s projects. To sind a 
group to vi*si,t the US'SR, 

Sketch. Using auch act ivi ties!, as a 
coronation and 'publication of a 
n^wspal^, the students ^became 
famllktr~Vrlt.ii nome^jRu^ijni^Lpa: 
;nut 1 \ iuv. 



expression to be refined through, ex- 
tensive studio work. Unders tandihgs 
of the freedoms affd disciplines or 
' art to be stressed./ 

Sketch: Each student completed 
several canvases. Themes such as 
' Orientalf religions and self-evalua- 
tion werW discussed. Personal gui- 
dance given. 



Comment: Excellent teaching, social 
jriencev Mitra, an. excellent 
facher, haci a b 



ge/ 
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iVmuu'Ht : M^^'u•h^M^*^lii>tl^^>otlMl^ , 

trlmi.: , \ tiffiMi I t-Mtr^pl*ut, the i' la 
M t*M»hm! i* 11 f.M*. ,r.Uitlt'iH . 



I ndll;*.- 

;t; too 



life, not apprec 
theless , 
one who demand 
of learned skil 
TOiTY-71 asset 
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hilosophv, of 
y ally^Never- 
sldence, as 

application 
s a major 
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^SESAME STREET I, II 55 students 

'f teters: Roger Clemerice, Tom Walz 
""nsioc: William Bichel 

* 

Aims: • An educational experience 
using the television medium to con- 
vey life goals and knowledge of ' 
basic social sciences to urban' 
children; with field tripb to gather 

• Ideas, role playing to learn per- 
spectives, script building to under- 4 

-♦. stand communication. 

Sketch: Two classes, one oriented 
to the stimulation of the city £ one 
to on-stag$ production."* . . ^ 

* 

Comment : 'Course^ilosophlcally 
elegant, operationally a mess. . Tea-_ 
chers personally sensitive but pe'da- 
gogically insensitive; so unwilling 
* ; to impose (to structure) that learn- 
. ing opportunities slipped away. 
Students turned on, will continue 
into fall. 

\ ' ■ / - 

/r* MATH FOR THE DISENCHANTED 22students 
h. Master : A. Indeliqato;>Assoc:G.Schef fer \ 

Aims: %o give the student who dis- 
likes- marth an understanding. of its 
^ utility and a look ax some inter- 
esting abstract ideas. Sample top- 
ics:. Simulation, surveying, topo- 
logy^, crystals . 

Sketch: Students decided to fibcus 
on two topics, surveyingfr'and geo-j 
„des±c domes. Computer math was 

added later. Students, surveyed Port 
- Snelling site to assist' archeology 
class. ; 

Comment: »£lass turned' out to be 
, more ^just curious" than "disen-^ 
chanted"' but rapport ^ motivation , 
.and involvement did grow. 

" MUSIC 25 stu3ent,s 

. ' Mas terr Jons. Reidel;Assoc : Judy Evans 

'. Aims: To increase, awareness oC 
r"*v classical, folk, and pop music; to 
•'V/^on^rast the music of tyortft, South, 

* and Cen.tral. America. -A*course for 

• developing understandings of music, 
.. # not, for developing the skills of the' 
o isician." 



166 io 



Sketch: Mimjeul simw 1 1 i y 1 1 y i:rrw 
with expo;uuv to ethnic inur-lc and 
ba&'fc eonoopto tu nuu* loo lory . Work 
with oiuior.il 1 1 oj* # - \ 

vVimnonl : v\mi r;-o com oj.it rxoo I 
. :'.UM io V:w WW I o;: IlWlttt'»|U;i t o . ■ : 

eontrUmtVii Mtllo \o other fluo 
'.arU courses op total Inr.t. i tut.o; no 

real tier, 'to Wooduloek Niitlon. 

WILDERNESS LEADERSHIP 36" atudentc 
Masters: Constance Gore*, Bob Taurine, 
Marc, Wanuig; Assoc: Ron Pressley 

Aims: To combine the skills of 
camping, canoeing, backpacking, etc. 
with the responsibilities of leader- 
ship and organization, so that stu- 
dents can guide others into the ... 
wilderness safely and witl) a sensi- 
tivity to it^ social, esthetic, and\ I 
ecological ^aspects. , Practical ex- 
perience. 

Sketch: Students practiced making 
•plans for trips ,• assisted groups on. 

Cannon River canoe outings, etc.', . *~\ 

learned the problems in , looking out 
jfor others in ordinary as well as 
^hardship and 'deprivation conditions, 

and constructed own pack frames and 

other items o ^equipment. 

Comment: A very popular and' worth- 
while educational experience, it 
should^be expanded for^forthcoming. % 
Institutes, but. should be changed to 
provide more instruction in guiding 
and wilderness living.* % • * 



SPANISH - ^9 students 

MasteriRamedoSaucedo;Assoc:VicBarela 

dlims: *>To experience Latin American 
TBte^on campus, in Spanish homes in 
ttf£^*** in Mexico City. 

Sketch: Activity Wiented, to Span- 
ish movies, Spanish home:;, Spanish 
kitchen's. 22 Students yi3i ted 'Mex- 
ico City. * * 

Conunenl^ Well managed; £laas suf-^ 
ferea a bit with the midterm de-' 
parturf of the favored few, but 
recovered* * (% ^ 



L 



1 r 

c 



GERMAN " .40 students 

Master Cameron: Assoc: P.Schweppe 

iims: To learn what the German peo- 
ple are like through a study of 
language and culture'. To send a 
small group to visit Germany. 

Sketch: Teacher worked on attitudes 
toward .language, "and attitudes to- 
. ward the work it takes to* learn a 
foreign language. Teacher -claimed 
. students developed a more sophisti- 
t cated perspective of the student's 
own culture and of the German cul- 
ture. 12 students to Germany. 

Comment: Associate teacher, left in i 
charge, gave us evaluators a h.ard 
rime. • , 

I ■ ' . - 

THEATRE CLksS ' 2? students 

Mastery R.beclercq; Assoc: M.Pfeifer 

^ Aims: TJap students in this class 
will prepare ahd participate in at 
least one dramatic production. They 
'^will get students from other classes 
/"to share their drama experiences, 
v- Story-theatre (Three Bears, Pogo, 
etc.) Will be developed. 

Sketch: Students put on RIP-OFF, a 
collage of story-theatre productions. 
Worked wi-th poetry groups to "con- 
trast and combine media' of expres- 
sion. . * - / 

Comment: Students worked frard, re- 
. sporided with enthusiasm and teamwork 
tp DeClercq's direction* the strong 
Internal rapport, love and respect 
~ for each other. 

r ■ 

POETRY ' 30 ''students 

MasterrJohn Caddy ; Assoc :Joy<ie Thomas 

i * j 

i 

Aim: "Imagine the human dawn. Ima- 
gine an ancient form of man;, prepar- 
ing for the hunt...? (So befcan the 
catalogue description of this class.) 
SFhe promise contained ways \tp awaken 
s > "the sleeping voice" , to egress 
c7\ oneself in "the ancient natural way 
'( /-wtoy". Th$ method: establish the 
» group, write what is personally ^sig- 
nificant y$ read recent poetry, talk; 
* i 

o ; r • • 
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. Accomplishment: Perhaps 3/4 of the 
3 students gained a. strong sense of 
trust in other group members, most 
did the writing -seen needed' as a 4 
•base for "sensory awakening", per- 
sonal expression. 

Comment: A most -sensitive, mature a 
teacher; students productive, some- 
what too dependent, with a not un- 
expected fixation on sex-theme 
poetry. 



ENVIRONMENTAL ACCOUNTING 25 students 
Master: Dennis Daly; Assoc :Mel Pibal 

\ Aims : To develop interest in accoun- 
ting and business; to portray their 
multifacted, dynamic character and » 

.the role they play in <^r society. 

Sketch: Discussions', simulation^"""" 
role-play ing^ field trips, a trip 
to Chicago. t Students learned how 
accounting is used to help indivi- 
duals and organizations attain 
their goals. 

Comment: Program supported by the 
Certified Public Accountants," State 
of Minnesota. Teachers ingenious, • 
one perhaps too verbal. 



ECOLOGICAL BIOLOGY 55 students 

Masters :Tony Angellar; Harold Strobe! 
Assoc: Bill Holmson 

Aims-: Through 'team teaching, to , 
pose and ponder questions, about the 
basic functions and reactions of 
animals and man, e.g., organismic 
learning, body-environment inter- 
action, chroriobiology. To measure 
and plot circadian rhythms, to carry 
on individual and group projects. , 

Sketch: Completed units oh animal 
behavior, learning, body-environmen- 
tal' interaction, chronobiology and 
circadian rhythm. . Completed fewer 
projects than intended../ - 

Comment: Great variety bf^field 
events; teacher-talk good talk but 
probably too much of it. , 
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5*2L' K M 35 students 

Assoc: Linda Nelson, Avi Davis 

Alma: To Involve oach tituamil In 
«ovi», C , U.l,,kI., l% ,iWM,„, :;it„;,: M 
tionb,; encouraging her I him) i,» i V lv 
on an expanding movement vocabulary*, 
an tweness of self and sensitivity" 
toward others, an Increasing know- 
ledge of dance as an art form and" 
Its relation to the other arts, a 
X desire to express herself creatively. 

Sketch: Improvement of self-Usaae 
and group awareness were seen as the 
I? 0 « c <> 0 »Plishments, with vari- 

• °£ cour8 «» across indivi- 
duals. Time spent m sustained 

ZI£hX!%?? dy T" , !? t assignments , 
watching films of master dancers,. In 
theme development, learning concen- 
tration, and developing -a viable 
group. 

Comment: Teachers highly competent 

v* nSlT 8 ; Jr lass needs b °y*» 

* de organised as supporting rather 
than as a principal enrollment. 



ASTRONOMY 



32 students 



Master/:Pred Brett ;Assoc:0«nnlsKallu« 

Alas: To examine the interdepen- 
dence of scientific facts as they 
relate to events of tha universe. 
To develop analytic and Inferential 
f*} 11 ** Individual, projects* parti- 
cularly building personal tef escopes. 

«Jj e !2?L4 L * rse blocks of **** apent 
a JFrit^t poliahln * lanaes. Used 

fS*v! n J : studenfcs took great pride 
in their work. Sustained involve- 
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WRITERS WORKSHOP .« t uludonU; 

wastef:iiuAmlt»n»on;At*ooc:K.K I opium* loli 

A law: To |tl Vt- ::liut,Mlti: Wl**-W:iill -I o- 
1 «••■■ ■»-• »VWf II r ;i rhattfo |«i |«.; in , 
• Miolr own w ,.| t |„ K . u . |rf| • 

>>w to s:,.;fivli r,.,- tiling worth writ • 
i iik ahoul . 

Sketch: A *::ulwoup rt f Uio elas:: 
pubiiahed %a Uouche^ tfie student 
newspaper; others in class wrote 

Irtllnl stjles"' 10 ^ 

Comment: A strange class, probably 
t^}?*!* iaaraing-oriented m the 
Institute. Students were not pushed - 
to produce"; many did not. Over half 

Sfv 8 i wdtnt5 wl8ned tn «y had en- 
rolled in something else— for the 

SSt wish? In8tltutc onl ? had 



S^^^., 58 stunts 

wasters: JSd Anderson, Pred Blaisdell 
Assoc ; Jon Gross, Virginia Toms 

Aims: To' lhtroduce new s.tudents to 
computer uses and programming. To 
introduce advanced students to FOR- 
THAM. To consider problems in num.: 
ber theory, geometry, economics. 

Sketch:. After learning the computer 
language, the students went on to 
learn many contemporary .uses of com- 
puters. They investigated the ef- 
fects of computers ott modern living. 
They learned, of new technical de- \ 
velopaents In the area of on-line - \ 
usage. 

Comment c Facilities crowded/ fre- 
quent trouble with lines ► Strontr • 
afternoon following. 
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TCITY-1971 EVALUATION REPORT: FIELD TRIPS 



Field trips were encouraged by bekg easily arranged: the tea- 
cher filled in a bus request form the day before his excursion, 
and the bus appeared at the appointed hour. Consequently, field ' 
trips were t often used by teachers to supplement classroom activi-/ 
ties and to take students to unique, learning sites. ■ 

' f Some classes were absent so much from the Macalester campus that 

• it was difficuU to decide whether they were on' field trips or whe- 
ther they were permanently based off-campus. All* of the Physical 
Science classes sfcent 2 weeks at Camp Tamarac. At camp, there was 
• . an integration of class activities into the Tamaraq environment with 

canoe- trips, nature walks, innertube floats and visits to Itasca 
State Park. Teachers felt that two weeks was too long. Students 
thought it about right, but one pleaded, "let kids go home on the 
f ^ek?nd and get clean clothes • " The archeology class spent 5 weeks 
r. m > e dig site. Three days in Chicago endqd the Institute for the 
~,nv ronmental Accounting class. The German and Spanish classes 
. < 12 and 22 students 'and a teacher -to Germany and Mexico. Almost 
-"ass spent from 3-5 days canoeing on Cannon River or St. Croix 
* ilh brigade leaders from the Wilderness ^class. (One day ^f 
in pe^ trip was about average.) 

Teachers felt .that the wilderness experiences helped to establish 
" * class identities.' Such field trips heightened awareness of differ- 
ent life styles, stimulated discussion, . and provided inspiration. 

in retrospect, almost all students felt that 4 these trips were fun 
and were good learning experiences. Other student comments were: 
itl "tpo much fun squeezed into too few days"^ "It was 'fantastic — I 
: loved it 11 * and "It should have come earlier to pull us together 
. mentally". Students evaluated their field trips as "mostly good", 
"better than time on campus", and* "extremely -stimulating" . 

In^addition to these longpr trips, the class activities calendar 
was punctuated with one^day excursions to places in the Twin Cities. 
Some of the sites and activities: Metropolitan Minneapolis-St . Paul 

* tour; Jonathan, model housing development; IBM Roches ter* computer 
facility; Pillsbury State Forest, wilderness; Apple River, inner- 
tubirtg; Teachers' hqmes; Metropolitan Stadium, baseball; Como Park, 
picnic; City Council meetings; Lino Lakes, model city project; Gen- 
eral Mills; 3M Company; Univac; Minneapolis Institute of Art; Isle 

- Royale, wilderness, St. Paul Ramsey Hospital, physical | tests ; New 
' Ulm, "German" town; Duluth, Urbary^Hero trip. 

Buses were expensive. * Students generally contribute^ part of the 
, bus expense and the cost of jneals. Butcsstuderits and teachers agreed 

that excursions off-campus contributed much to TCITY-1971. 

f Craig GJerde 

Evaluator, CIRCfc 
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TCITY-1971 EVALUATION REPORT: THE' OPEN CAMPUS 

tils t h ° , M, sn3. , , r „-v'xi, , ^,r. fc J iv,;- 

pleased. pune8, . Most felt tree to i>ome and gt> a« CTu»y 



-^"SSu-SSuTSl-Sll^S ? r ^ ■»»«"*•• They 



T. Denny 

Evaluation Specialist, CIRCET. 



. An Adversary's statement-continued from page 2? 

Pew students— or faculty— understand the selection 
procedures employed to staff the^teaching cadre and to 
fill the student corps." Why should it be a mystery? 

The worst has been saved for lac-t. This' report concliides u n-h 
an- assertion:- the absence of a crucial dimension in the Inst™- 
„ tional life of -T-CITY, that of constructive self-criticism ill 
-near fatal flaw. The. observation and interview notes taken by ?he 
adversary evaluator over four days eoqtiKns but five Instances of 
students engaging in, oxr*faculty helpinVstudents to b2?oSS Sill- •> 
SSJS.? P £ e£ 4 rou * of > the cultivation of self-criticism! The 
instances or missed opportunities were excessive ia my ludWnt 

5tt2 eShSsJa^fV* 'J V^er'aculty^nd studenVa&howed 
In»tittit2 SSSfX r SU ^ fare * 13 ifc too much to expect, from 

!k P i 1 5i pants after but ' fQur weeks? Seven may be insuffi- 
ouTto start 3 :f t f he P °:nd? 0rtem ' > ~* * st^Uut^fs'eem^l. 

' M n75li? aUC JI y ° r occurenc * 13 less damning than the- absence of . X 
jianifest, widespread intent. Certain classes accounted -for- all 'the 

N instances observed. They did not appear to be accidental! The in! 
^ < ?? S K 1 the ^; n An In stitute for talented high school youth ™?nnSt 
justifiably fail. to feature individual and group self-criticism? 

' f/,rn ( £ re P ared b y ?• "Denny'* not to indicate his opinion of TCITY- 



tcity-1971 evaluation jjeport: STUDENT ATTITUDES 



The Director and Evaluator tried to keep- in touch with student 
feelings. Once a week each student was asked three questions taken 
at random from a pobl of 12 attitude 'questions . Thu^ each question 
was answered each week by 75-200 students. 

Four,r quest ions pertained to student learning. Every week at 
least 90% said they were learning a lot. The number of students 
saying TCITY is one of their best learning opportunities increased 
steadily from 75? to just under 90%. The number who reported them- 
selves involved in creating or developing a project increased from 
.405 to over 60% by the third week, and leveled off. To the question 
"Do you have the feeling that — when this Institute is over — you will 
say that TCITY has been a very satisfying experience?" about 80% 
said f, .yes M until the final week when the positive response rose • 
to 1005t. ' 



Throe questions pertained to 
how well the, student, liked the 
Institute. One is shown at ^he 
right. Everyone responded each 
week that they liked the people 
* at the Institute. About 75X 
wtM'e satisfied with afternoon 
art 1 v) t leu though the percent 
dropping toward the end. 

r t At' first, about naif the stu- 
V. dents ^ald Lhey were getting 
too little Information about 
TCITY events; by the seventh* 
week a quarter still 3ald so. 
Another administrative question 
is shown below-right. Another 
brought forth^e almost unani- 
mous opinion ^that to appli- 
cants, TCITY offers a selection 
of challenging, relevant and 
useful courses. 



Are you enjoying yourself at this 
institute? 





Consider the whoJe Institute (and not 
just your class) Hew the arrangements 
been well planned end effectively 
carried out? 



po&riy plenned and carried out 



well plennedjpd carried out 



i i s i s i 



12 3 5 6 7 
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tci.ty-1971 evaluation report: A' PARTICIPANT OBSERVER'S PERSPECTIVE 



As my part in the evaluation of TCITY, I was a participant ab ser- 
ver. I was a member of the Wilderness Leadership class and wa:; in- 
volved in the guidance of other clajjsei-. on canoe and camping trip:;. 
Having had a fair amount of *proviou;j camping experience I war, nble\ 
to observe carefully the learning experience?:; of other ntudenU;. 

It is apparent to me that 4 student:; want to learn.. At TCIT^v^ 
teachers provided the opportunity for an interesting learning ex- 
perience. Students took the initiative, and "away she goes". 

* 

Students will learn, even under adversk c c injidTn^ww*. . On one oc- 
casion after a nearly sleepless night Tmdfijv^Snon-rainprbof shelter 
a friend came to sit-out the rain under my dry shelter. He was in 
low spirits and explained to^me that he was* not learning \what he 
wanted to. He said that he was learning what was wrong, but' not 
what was right. But we figured out his teachers were not trying to 
teach' him what was right. They were not trying to teach him the' 
answers — they were trying to* teach him the questions. And even 
though he ^said he didh't want to, he was learning. They kept put- 
ting him in situations where he*would learn. 



I believe the Wilderness Leadership class was a very successful 
one., The studejits could have tfeen taught more than they were, and 
they could havV learned more than they did. But they enjoyed it, 
even with adverse conditions, and learned about handling themselves 
and others in the outdoors. Lots of time was spent not' being stu-' 
dents, just*be,ing thera&el*ves . It seemed they learned just as well 
that way. \^ 

One of the tirfiportant reasons for 'the success of the-class was 
the freedom given the students the freedom to choose -whether or 
not to go to class or — in. triis* class — which trips to go on. With* 
the amount, of involvement that was asked, at trip almost every week, 
it would have been easy for the students to reject it. But it was ' 
their choice,, and they were gla,d to do it. . 

The important thing in all the Institute .classes I worked with 
was the atmosphere that influence^ students' attitudes. The atmo- 
sphere at TCITY was one that encouraged the student to learn be- 
cause he, wanted to, not because he had to. 

* *■ * 

The*basic* elemenfT"of this atmosphere was freedom:* freedom of N 
movement in and out of ,fche classroom, and freedom of dhoice of sub- 
ject material. At TCITY I saw students, moving, choosing_ learning. 

* Ben Stake,. Student* 

University High School 
Urbana, Illinois . ■ 



Welding torches 



Gopal Mitra 

TCITY-1971 Art teacher 



( 



Oip^Tninds are welded Into 
a useless facts Index 
that swells In schools. 



s^r 



Offered a torch 
we Ignite the 
f . stagnated f^rms 

* sculpting our minds Into 
geodesic domes 

and polyethellne bubbles. 

* Teachers/friends light 
the torch 

but we shape the sculpture. 

^ Ignoring tired eyes 
_ a-leeple^d faces 

• arrive and awakens 



v 



Life's paintbox- 
colors our experfence 
but we find It 
here.* 

<Wriy mu:it 5 »H dry* up - 
j>nd !>Lter come fall? 



iits 



Be>tsy Mars (Ten 
TCITY"-1971 Poetry Student 
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TCITY-1971 EVALUATION REPORT: COLLECTtl^E PERSPECTIVE (students) 



r 



'The following was taken from the tally of tlw» Student Final 
Critique Sheet.- About 560 students fllUM out the ;sheet during; 
the final week of t^e Institute. 

We would like to know how the Institute class activity differs from the activity pf the 
classes of your regular school* 



No 

Difference 



Regular 

School 



1. Which holds students more responsible for work? 
2* Which encourages students to "show oft 11 more? 
3* In which do students try out OWN ideas more? 
In which is more time wasted? 




And how do your teachers compare? 



5# Which explain 
6. Which know the 
7. 
8. 



hings better? 
subject matter better? 
Which understand students better? 
Which resist the urge to talk all the time? 



No c 
Difference 

42* 



Institute 

20% 



School Institute 
Teachers Teachers 




Preaae rate the following features of the 
Institute as to how important they were to you: 



Extremely 
Import ait 



11? 



A little 
Important 




Close contact with teachers 
Close contact with students 
Opportuhity- to study for extended period each day kCt 
Trip to camp, canoe trip, etc. ' ^6% 
Exhibits, performances put on for "outsiders 11 
Diversity of students 
Afternoon symposia 



m 

60% 
28 



ERIC 



27. 
28. 

.29- 
30. 
31- 
32. 
33. 

y*. Concern that these people have for human problems 69* 

35« Faith thatvthese people have that these problems 

can be solved^ 69* 

36»- -Being treated ias a mature person " 85* . 

** Here are several goals of the Institute. Please rate the Institute on Jiqw well it met 
these Tubals, even if you only have a vague idea of what the whole Institute was doing, o 



37. * Provide an educational program that is. 
. challenging, stimulating, relevant 

38. Provide master tead&ers with the highest abilil 
to-teach * - 

39» Provide younger teachers a good opportunityjtto 
learn more, about teaching* 

*K)* Develop curriculum ideia that can be used in 
18 the regular schools* 174 




Excellent 


Barely 
Passing 


Failed 


Don't 
Know 


_86£ 


10* 


0*. 


JL 


- > 

81* 




2*» 


-21 


69* 




1* 


16* 


■67* 




3* 


17* 
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tcity-1971 evaluation report: COLLECTIVE PERSPECTIVE (faculty) 



In the final week of tlie Institute all master and, associate tea- 
chers were asked to complete a four page questionnaire. About 
3/4 of the teachers did so. ^' 

The master teachers listed the major satisfactions *in UCITY-71 
as the opportunity to teafcj^the way they wanted to; to work with 
highly motivated, able students; and to make the learning-experi- 
ence a humanizing encounter. The associate teachers emphasized 
these things plus the fact that the Institute was a major learning 
opportunity for them. Both groups were strong in their praise of 
TCITY. . 

The least satisfactory aspects of the experience these teachers 
hacKwere the administrative arrangements (too vague, 'too little 
pr£-planning) and the workload^ ( too much „exp$c ted) . Some master 
(teachers' objected to the large number of students enrolled in 
their classes. Almost all the associate teachers . reported some 
unpleasant interpersonal experience during the sutamer. 

Special features drawing^ approval were the May orientation ses- 
sions and Wilderness trips*. New arrangements that teachers, ob- 
jected to were the admission of junior high students and the 
foreign language travel alternative. * 

Among the suggestions for improvement were the following: 

1) Increase involvement of University. stu- 
dents as teaching aides. 

2) Better communication about T£ITY events, 
deadlines. 

3) Less involvement in formal evaluation 

* 4)' Extend the opportunities to- suburban and 
out-state students. ( 

By and large the faculty responses in 1971 were similar to what } 
they had been in 197*0. There seemed to be better communication 
across faculties in 1970 but better communication to and from the 
Directors in 1$71. Communication remains as an important problem. % 

Most of the master teachers *were 'pleased with wh£t they had done, 
but still would like to offer # a different course or teach the 
coiAe differently next year. They strongly hoped that TCITY could 
belHcated at Macalester again in 1972. 



Jetail their activities -for the summer and to reac*t more 
thoroughly /to such evaluation issues, each 'master teacher submitted 
a synopsis called by Caruson "The . Gleanings" . The&e reports were 
loaded with recomipendations for summer courses* and institutes. 
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tcity-1971 evaluation Report: EMPHASIZING COGNITIVE TALENTS 



In a sample of about half the closer., re^porn-.e* to the 
Activities Questionaire (CAQ) indicated a few things about I he 
individual clasr>e;; and the Institute an a whole: Kvery el:i:-.s 
showed much more emphasis on the higher level thought proce'::- 
es — application, analysis, synthesis, and evaluation—than on 
the lower level thought processes — memory, translation, and 
interpretation. 

Every class indicated that ideas were valued more than grades, 
and that there was much humor and enjoyment of ideas. There was 
very little lecturing or test stress. Enthusiasm, independence 
and divergence were indicated in each class summary. 

^ ' Students reported the percentage of teacher talk to be be- 
tween 10 and 75 percent with the median at To percent, 
r ■ - 

The CAQ profile for the Urban Hero class showed these responses: 



Levels of Thinking 



V 



Lower 


level 




Memory 




- none 


Translation 




inco/iclusive 


Interpretation 




some 


Higher 


level 




Application 




some 


Analysis 




some 


Synthesis 




inconclusive 


Evaluation 




some 



Classroom Conditions 



Discussion opportunity 




much 


Test/grade stress 




, none 


Lecture 


* 


none 


Enthusiasm 




, some 


Independence 




much 


Divergence , , 




mucH 


Humor 




much 


Ideas valued over grades 




much 


Enjoyment of ideas 




• much 



Percentage of teacher talK was 40* and average preparation time 
per weejc was 1 hour. 
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t^ty-1971 evaluation report: EMPHASIZIN6 CREATIVE TALENTS 

v . . .. . . . \\ 



T~*~\T ■ VK-r'' 



Question: What at A Twin.City fosters Creative E^p^ession? 

H.,Jf h(? - f5° V ? a ? d ^'"i 1 ^ Queries were asked during two morning periods 
during; the last ten days of, class. Primarily, small random groups of - 
kinds Were v questioned— about 50 total— NOT in the presence of instruct- 
ors. Their areas included painting, sculpture, poe try ¥ writing, and - 
drama. Also briefly ."interyiewed .were two instructors in sculpture 
and dance. 1 The choice of students was based on availability J Also, •* 
two eve/its were witnessed in tdrama^ arid dance, plus* paintings and 
sculptures were viewed as works in\pr<5gress . - ' \ 

Answers from students- included^ « H ** *. 

"I really enjoyed the canoe trip.?* o* . . , 

"I got a chance to make a lot of friends." «' * <v 

"I am having a lot of . fun." • 
"There is freedom here, we can do what we want." 
"I learned a lot." 

'•My regular school catagorizes ,* here we can switch out -Of areas We • 
aon t like, and extend time on projects we do. like. "*" - 

. My response to the above is that they have a different set (from 
Si2?i<?f id !v S ° f wbat con stituted Creative Expression/ Within .their 
definition they seemed to succeed. - They suggest '.that ^success of the.- 
rwln City environment to foster C.E. was- • "' ' 




*• IV l he milab ^ 1 } t y t00l s and teachers— NOT the enforcement. 

r 2) The availability of manipulatable time— NOT short class periods 

• - and semester-long courses.' ' ' 

• 3) Special community experiences, such as the canoe trip. ' 
4) Instructors who were their frJehds.. ; '° : t 

. . Other observations revealed that in the areas of Dance and' Drama 

;(two reheaj-sable arts) physical and communal coritacts imbued kids 
<4 . , with > special sense of C.E. Here, students were'glven a chance to, 
.note day by day growth* -as well as. longer range 'improvement measure- 
ments. They were appreciative of aspects of process'and product, 
(learning and. final performance.) They were given a chance^ 
1 pret and develop. (always within . community approval). This 

true of Poetry and its readings-.* 

t 

Personal Remarks: - 

■ < ' ' ■ ' \ 

"I raus't state that ( the student notion, of Creative Expression is 
vastly different- than mine in many<ways.- Their definitions move 
toward C.E. as THERAPY AND/OR SOCIAL CONTACT (global village concept?), 
but f rpm my experience Creative. Expression' 'is^less .provincial. Pgr A 
. me , it. Involves larger, natural, "gran^r^thingV: ' interconnection, 
► growth,, mutability,' choice, discovery th> fullest mulfi-level 

rxper loriirea. • ' * ' 

• ' ' ' - ' ' - . >v . \ • • • 

* * 

• >u»t r.wiv l hat t.hjn Is possible in a seven week program." 'ButV" 
• HVIn City mu.-.t kiM»p trying, with a faculty that exudes, inspiration as . 
It iJt:-.;-.«MnlnatVy Pact;* ami trains skills; * nurtures Imaginations as the 
_ tt«t|ul , |V I law:; I ho in I ml into- the ooc-.mos." * ■ 

# 9 * 
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tcijy-1971^valuation report: THE LIGHT-H0USE*§tH80L . 

with Institute teachers and social studies department chairmen. 

n^^if es P eclallv appropriate- to ask if, social studies teachers 

"J a " OC * ated Wlth the In ^itute are affectel by ??m 
- offerings in social science. This field involves virtually all 
; secondary schbol students and. some 300 teacher! in the district 
Furthermore, dissatisfaction with current sScIL stuSles p^o^s 
,has been expressed by students, 'teachers and administrators SI 

pr6a?nes ren ?n iZll 5* con ™<"ing £.$5™ Sid apl 

ImortSfen ' !I* e ? 5 o r chan «e is felt, decisions 'aboKt 

rY* 0 .- t : P. fleld must be worked out locally, and the Twin • 

• 3i$*:i 1 5! tltUte has enga * ed ln structuring learning experiences 
that might serve as models for new social f tudies .coursel. ' ' J 

^hJ^^^^^f rec ' 6nt TCITY work ln th e socfal sciences suggests 
*here would be merit, in public school considerktiSn of Infti^ 

To XML ^ ?tr ° ng P ° lntS ^ the TCITY ^oSid^SJar ' 

>• - '"w?rnnL^ d K teaC 2 e -Tf nave act /ely explored the Urban 
<s environment beyond the classroom. 

" fS^ G ^ a i attention has been glCen to student interaction ■ 
and interaction between -students and teachers. 

' » " deaIln ? ' wlth values have been sensitively and 

deliberately explored as an integral part of the learning ' -• 
Of concepts and skills of social inquiry. . g 

*• Sn?S t S el * SS \ tn ? lmpact 0? these ef ^°rts in Minneapolis high schools 
would appear -to. have been negligible -because of the following factor! 

•SStSS ^^rscJooJsr aPPr ° aCheS afff ^sMfferen P r?ro e ^th^ . . 

aualftv 1 ^ ?SSf?i rt EI^ 8 S ne } acknowled «5 e the "special" and valuable ' 
' have f«£T22 ?n«,?^ e ? ln8S - H ° We y er * a^niatrators and teachers 
dSn!'?n !?! In ? tltute as providing models of what might be 

is" ^iJfJft c J- a ; srooms ; Teacher knowledge of Institute offerings 
• i!J?5h!5ifc a !S lnt ? re !\ ln furt her information has not been exprefsed, 
: cen?e£s? interest in programs from, curriculum development 

^La vE!! 1 ^* !f5 C ?? rS tend t0 view thelr TCITY experience positively 
' 8£ J5L thOU 5 h « tfU iS eXpl0J?e the ^Plications of this experience ' 
Tor their, work in the' schools. The autonomy? and isolation of their 

•. '■ • 

22 • / ; I tO 



regular teaching situations does not provide the exchanges that ■ 
would encourage tfeem to assume leadership in the reconstruction of 
departmental programs. Several of these teachers are concerned 
about their inability to communicate their greater awareness of the 
complexity* of teaching-learning processes to colleagues in ways that 
would further their own reflections and growth. The resulting 
frustration would appear in a few instances to have contributed to 
"decisions to leave 'the regular classroom for graduate study and 
other, positions in education. > * 

■ ' . ^ ' ' < ■ , \ f 

Students who have been, involved in the Institute apparently do 
not return to their schools as agitators for change. Students., ^ 
like their regular teachers', apparently see the Institute as a 
unique experience separate .from the standard school program. It 
- might be hypothesized that these talented students have learned 
•how to be successful in the public school environment and they 're- 
sume their ..previous student roles when they return to the settings 
in whloh .these behavior patterns have proven successful. 

6 ' • ' \ . ' ■ "4/ 

It would appear that jfche culture)of the- schools and the unique 
features of the [institute tend to promote the. separation of these 
experiences. Special programs like TCITY are isolated from the on- 
tn£, business of the schools because, of expectations that students, 
. ctiool personnel and Institute faculty have for both the public 
.uools and the Institute.* Explicit efforts to use the Institute's 
fe;c/eriences as models for* improving the public schools must be de- 
• V£aed if a development *nd demonstration- function is desired^ Even 
then success cannot be guaranteed.' \ Efforts .to use the regular Min- 
neapolis summer school as a place to .test 'new ideas and to serve as 
an inservlce laboratory fpr curriculum development . and, the adaptation 
of materials- from national social studies projects have met with very 
limited success. 

* • ** « 

If the Institute Is to serve a more effective role in dealing 
with the' urgent need for innovation in the social studies- and In 
other curricular areas, detailed planning Involving a fringe of * 
person* — Students* teachers* administrators, chairmen, arid .con- 
sultants — must take place v It must be clear that any effort to 
use the institute as a development and demonstration center* for • * 
curricular revision-might very well compromise many of the unique./. m 
cfualltles now Included in the program as It seryes the special needs 
of talented^tudents. » * 

Again/ it must be stressed thtt these are personal observations 
focusing on social science programs* In those curricular areas such 
as computer mathematics the TCITY experience. may be a significant 
element in a broader Innovative effort. The Institute . teachers of 
computer .math also teach these courses In their Schools and they in- 
struct other teachers In staff development 'courses aimed at estab- 
lishing similar courses. Also* lh foreign, language and in the . 
sch&ols involved In TCITY Instruction and their work has. continued 
Into the school year in ways that have sfgnif lcantly influenced their 
departments., However, the required social studies courses Involving 
large \numbers of teachers and all students In each school have not v m 
felt the impact of the TCITY experience. 



-Robfirt Berry, Cur iculua£ Supervisor 
Rochester' Public Schools. 
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TCITY-1'971 EVALUATION REPORT: *" MANAGEMENT A 

If they* changed TCITY to 'an All Minnesota Institute for Tal- 
ented Youth — as some have suggested — they could use that welll-known 
slogan, "You're in goo,d hands with All-State. 11 TCITY is ahapej by 
■ able" hands • ' ~ 

The authority of the Institute is highly centralized. Charle/ 
Caruson* makes the decisions. .The teachers have a great deal of 
leeway; they are expected to take bold steps. But "the Director 
Man" is in close touch with what they do* He's the aate keeper. , 

Charles Caruson tajiks a lot. . And he listens ^^^srx. And he askA 
, a -lot of questions. He strides across campus, abruptly stops, c 
flashes a big smile, and asks a student a pointed , personal question. 
"Are you going to the Batique Demonstration this afternoon?" "Are 
you still riding the -bus to campus?" "Did you have a good time at 
Tamarac?" 

' One little girl coming into\his office early in th& Institute was 
surprised ,to see him behind theNt^sk. She said "You, the^ Director 
Man? I thought t)he, Director Man drove, around in a big black Cadilac. 
Carusofc does de/y the image t * 

Bob Rose is/second in command, the rest of *the command. *He maked 
.many of those sleeping rearrangements resulting from Caruson 's ne- 
gotiations. Rossis a hard, productive worker; his track record is 
impressive. He and Caruson work together sensitively, spending lit- 
tle time together, each knowing his responsibility, doubling up when 
the situation warrants. . ^ f * * t 

Miss* Stepanick handles^the secretarial chores in a no-nonsense, 
get-the^Job done^way. She gets* something reasonably productive put 
firf her summer staff of amateur office workers. There is no obvious 
substantial fault in heir office operation. (No effort was made by 
this evaluation team to* check on fi'scal integrity. Also, by appear- 
ances, Caruson and Ro^e do a good job of making contacts for finan- 
cial support — but this too was not examined..) , 

The" style of management of the total Institute "is more intuitive 
than it is deliberative.' Tftis helped make the 1971 Institute more 
immediate, sensitive, and flexible bu£-less exportable ,( to other 
districts) and less of a firm structure against which individual 
teach(ersf can £lan and' operate. For example, if the teacher doesn't 
knbw;the special all-?Ins titute events for'tl^e next week it * is 'diffi- 
cult to plan special class activities . \ ^ „ 

The director and associate director deal -with staff, students, 
and, Visi,toi:s in a casual manner. Caruson/s door is open for people, 
to walk in.. Many do. There are bound to, be some wh6 wish he and 
•'Bose Kould manage things in more conventional ways. J Such people 

* are more likely to remain sclent; we d,id not fiJid any. 

* \ Th'e mo^t impressive ability of TCI management is its ability fco 
. provide* outstanding master teachers. V 4 

Overall' the TCITY management gets a high rating for employing 
high quality teachers, keeping red-tape at a minimum, an'd being, . % 
sensitive to stfuaent needs. It rates a low mark on utilizing uhe 
potfenfcial oT\the Institute for district "staff development 11 and cm 
keeping ^ Ins y^ute people informed as to what others there are doing. # 
To the eyfeiWl 'that- these latter are outside the purpose of TCITY, 
the management is seen by the evaluation staff to be outstanding,. 

' h . -/ISO \ - • ' //'*. 
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TCI ty-1971' evaluation .report: MISCELLANEOUS ITEMS 



* Information Flow , "Getting the word" continued to be a problem, 
£ . ^ 6ut not as much as it was in 1970/ Bulletin boards, news sheets, 

^ "memos and grapevine carried 'a substantial information load — but many 

( , students and teachers — perhaps a third — felt, uninformed about TCITY 
y happenings . 

■ ri 

» Productivity , TCITY teachers' appeared to emphasize a project or 
<\ performance more in 1971 thah. > 1970. Still, only about two thirds 
of the 1971 students got involved in something they would call a - 
project. ' % _ 4 

Teaoher Training . TCITY offers the observer an excellent array 
^effective- teaching styles. Trainees can here find*" opportunities 
to assume teaching, responsibilities . (55 *U of M preservice students 
made d^ily observations,; some took as much of an assignment with a 
- class r as the Associate Instructor.) Still, teacher training must 
be considered a minor . payoff . Too few teachers are involved. Many 
more would' probably contaminate the studertt learning experience. 
If teacher training were *t,o fce considered a ipajor TCITY goal it s 
would require substantial allocation .of resources to. this purpose. 
If TCITY were to be used in any substantial way by the two districts 
for staff develdpment , a better way of rewarding experienced 'teachers 
for participation would have to be found. 

Afternoon program . The af ternoon/program was better in 1971 ,t#an 
/""■ * 1970 — in .both quality arid quantity. However, less than 30% of the 

* students were involved, even on one^ of the better days, in au TCITY- 
1971 afternoon activity. The 'students were not upset^ — they Just 
had* other things to do*. 

Hostility . Stud^nts'seemed to 'be less hostil'e toward staff §nd 
; each other this *Summer. The pleasantness of the campus and the' wild 
weafcher may have made a difference.* - Tt^ere als|> may be a lessening 
of* the. aftermath of Kent Spate.; « * 

, ■* Canoe' Building . Science students were expected to enroll in a 
second subject,^ a mini-course. Most mini-courses were science' 
cours-es, bir^ the most spectacular was a course in canoe-building. 
About a dozen students built handspme redwood canoes for themselves. 

Recommendation for 1972 Evaluation-. . The questions raised by the 



1971 evaluati 
■staff in 1972 
evaluation st 
that dlfferen 
of evaluation 
operating bud 
high ro;;T^tan 



oK team are likely 0 to ^continue to influence the TCITY 

The staff would probably benefit by engaging an 
aff with a rather different orientation in 1972, so 
t problems "will be identified and studied. The cost' , 

should remain at (or drop below) 2% of the totai • 
get. The next evaluators should anticipate continued, 
ce' from' many ^teachers and students./ * \f 
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TCifv-1971 evaluation, report: AN ADVOCATE'S 'STATEMENT J 

' f^ H i* m VlS l t0r : " h ° t00k - a Lon &> harcL iook at TCITY-71 kept his skep- 

to h "r g v ^itor knows how precious it is to dis'covir, t^be ' 

wheS'teLhei? !Ah S ld ? r ViSit ? r kn ° WS the rarity of a classroom 
where teachers and students perceive each other as real people. To 

can ^7v^ it0r 't? d ° eSn,t jeeB > PO«l"e that a summer , ho> roj ,„, 
can deliver on all these promises to ovor 800 kids, but TCITY- 71 1 
Every curriculum specialist fears that by relaxing conduct rile 
and encouraging student independence they may be saying goodbye Jo"' 
the hard wdrk and hard thinking that education requires TCITY 71 

IwarSe"fS? th'H a 6 ain and again to curricular themes: 
«^^S^J!TSc? ,lldltlOB - 0hStaCl ^ t0 —"Nation, ecolo- 
_ TCITY excels because of its staff. L*s students give it movement 
Its directors give if nurture.' Its teachers give it movement nur- 
ture and -direction. It would be incorrect to say that SJ Carlson 
Mr Rose, and the 'teachers think alike as to the prime goals arid 

' 52m;c2lll d ^ t tl0n ' S ufc . co ""»vely. they m £VT^Sl e ^ 
manistically-bent, -academically-based curriculum. 

day to daf 1 rJ„S f ,f eaCh J ng f hlS SUmmer WaS con sistently high, from 
aay to day, from class to class. Some of the teachers chase to be 
•casual, to offer "opportunities", to share a meaning?!*^^ence 
Others were more intense, more intent upon sharing in?or£ti& 2nd 
problem solving method,s. Both kinds were there, doinf 
«. I T 5 e j ( l uailt y of ,the learning" also was high. The stUu, 
tuned in. They were- busy. They responded to the moves ^ 
teachjrs., They improvised, they carried^Ldeas and argyuments //in- 
dignations and admirations, to the voL^baV court, 4l-thVcW>ns 

tooi h I S22%?f cam P us / lms ™« Cahn^Ler\aks. 't^o^s^s ' 
" * long step towards maturity. // \ X / 

dollSl* ? COStly St6p - Tho #ands \r houV^housands of 

of IZZlL i ! lj t a V a few hun 4^d\aggrav/tions. But fit to a scale 
^ P !!i ? 1 bud g et #--and budgetKjo^arks, interstate highways 

SLTK? ?Lv^'" TCITY - 71 rates as a BEST BU *« 800 kids, give ' 

a!ir*L« \ h ° me ! flew talent » a ne * * lne of.thinking/a^ew - 

awareaass — a good purchase. 

P-ui^f™! = 6 denie ? that ° ther y oun g st .ers in Minneapolis and St. 

slv SESJ if »l?Z ri T e :i ke this - They should hav * Some 

say, TCITY is bad because it caters to, the elite." But a ereater 
tUtZ ? ayS " Any effQrt flxated °n giving an eqdal sh^re Osgood 
thing, to all groups Is destined to share nothing of value." For 

fc«n?hff Van ^ g ?^ l° Uth ' a m °^ e ec l u i tabl e share of educational oppor- 
tunities should be guaranteed. But even in times of economic re- 
CeS mir^ °PP° rtu nities for the talented should be protected. 

TCITY-71 has succeeded. It is even a best buy. It*satisfies a 
social obligation to specially educate some of those who will lead— 

TCI^%f 111. ^Sf 5 ' in government, in life. The £eachers of 
TCITY-71 have blended a summer of caring, caprice, openness and in- 
tellectual struggle -to give potential- if aderfa* summer 4*anenge\ 

tut* ( ^ r f pared by - H 4 Stake* not \o iron-at* his opinion of tho i n . ;tl 
SS'bS made.,) 3U -* r ™ the f ^ clalnj tUt might 
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TCtTY~197r evaluation rerort:' AN ADVERSARY'S STATEMENT 



T-CITY is not a scandalum magna turn . But it i,s .both less than it 
pretends to be* and more tnan it wisneg to be. \ There Is enough evi- 
dence at least to question certain facets of the Institute — if not 
to return a true bill against it. Costly, enlarging, innovative^ 
exemplary: these Institute attributes 3re worthy of critical exami- 
nation, * . 

How costly is this Institute? Doilar costs are sufficient to 
give each group of six students $r,000 to design and conduct their 
own sruiran^r- experience. Over 100 Upward Bound students could be 
read led yf or their College careers at Macalester. About- -twenty five 
expert curriculum specialists could be 'supported for half^ year 
to design and develop hew curricula for the high school. 

What is\the cost of removing 800 talented leaders from the local 
youth culture?. What is the cost of widening the experience gap be- 
tween Insti^ut® stCsdents and th'eir parents? . and their teachers in 
'r-eular" high Yschoda? . .\ and theinr non-Institue friends? Nbt enough 
ert to -charge rvep-Facist elitism. Enough to warrant discussion* 

Inst/tute abounds wi^fi^self Miamed innovators and Innovations, 
> .x.tei^tatives -to the t^ifsiness-as-usual education of high schoolers, 
tic*' raat/the Institute is not promoted as an exemplary alternative 
o schooling. It seeks tp prQmote the development of alternative 
brms of education for schools* And it Is failing to do even that 
Job., Wh^t Is T-CITY doing to demonstrate that the T-CITY style of 
life coula-fee--4J^3x4Jri ^<5nools as we know them? Where ,in the regu- 
lar school Is the staFf so crucial to the life of the Institute?.*, 
the money?... the administrative leadership? Where are the opportu- 
nities for the teachers, principals., superintendents to come and 
live that life that they might come to share in the vision? .. .and 
where are the parents? *T-crj!X- should be getting poor grades on af- 
J^irtng the regular .schooJ^program. 

There are other dimensions of' T-CITY that puzzle the non-believer: 

## * How long can in-class "rapping", continue and still 
qualify as educative self-exploration? Are there 
^quality control procedures in effect during the 
summer program: For example: when one-third to 
one-half a class is absent from a schedj^led meet- \ t 

ihg, should not that be seen as an educational 
"crisis by the instructor? " * % 

- *** What does T-CITY dp to help students^ealize that 

• the Institute standards fere necessai/ily high; that 
the regular school norms and expectations do not 

• count; that a heretofore "best" becomes just a "so- 
so"? There are unnecessarily disheartened students 
in T-CITY. 

*** U; it imr.rasori.ilrlt* to oxp*>ut that more than two of. 
\ w<Mrty-two teaiMuM 4 :- or- dissociate teachers would have 
;unio clear 'idea or plan for utilizing "T-CITY approaches 
"£* rn virrii'ul i -Jn Mu-ir regular classrooms next fall?. 

• j % 

^ntfnued on page l 1 * , . . 
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^Talented youth' in perspective 



. The controversy over the archaeo- 
logical excavations undertaken at 
Welch, Minn., by students from the, 
Twin City Institute for Talented Youth 
(TCITY) has focused attention on one 
part of its curriculum* but has over* 
shadowed its total program. And 
thafs too bad, because TCITY is 
considered by many in education to 
be one of the best and most innova- 
tive summer programs in the country, 
and it points toward ways to make 
regular " schools more effective in 
meeting students' needs. 

The institute was begun in 1967 in an 
effort to get talented students from 
Minneapolis and St Raul schools to- 
gether so that they could challenge 
aadhearn from one another and de- 
velop their abilities beyond the lim- 
ited, scope of the normal classroom. 
The curriculum included - advanced 
course* not ordinarily found in school 
systems, but the approach to them 
was along the usual lines at first, with 
each student sticking to his own sub- 
ject area. Since then, it ha* evolved, 
both I n form and content Last year, 
TCTTY organized all students' pro- 
Dund a theme, Man and#his 
and encouraged an in* 
y ap£jp6ach to that gen- 




This year, the^heme is Man and So* 
ciety. The archae&iogical dig at Welch 
was one approach to that theme, and 
the institute's catalog describes its 
premise this way: "By working with 
a culture outside your fcwti, you 



should gain a different viewpoint 
from which to not only view your, 
own culture, but gain an insight into 
the uniformity of man, bpth tempor- 
ally and geographically." 
* 

The rest of the curriculum, whether 
in the fine arts, sciences, languages 
or social studies, has taken that same 
human-centered approach* Students 
have been encouraged and helped to 
become aware of and to learn from 
the people and places in the com- 
munity arbtmd them. One course, for 
example, is Called "The Urban Hero,'* 
and the catalog points out that "the 
hero* is happening in the city now" as 
people learn to survive in the urban 
world. 'This summer will be spent 
finding the urban heroes — the artists, 
. the activists, humanists, philosophers, 
urban guerrillas — and learning the 
politics of survival." 

Does it work? That question is per- 
haps best answered by two partici- 
pants irv— fast summer's • pirogram, 
quoted by Tribune staff writer Cath- 
erine .Watson in an article Amer- ' 
ican Education, a nagazmg published 
by the U.S. Department of Health, 
Education and Welfare's Office of 
Education: 

"ThisTs the only school I know of," 
said one girl, ,4 where you get dropins 
instead of dropouts/' And another 
said, "When you see what school can 
be like here at* the institute, you feel 
an. almost desperate need to change 
the schools/* 
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